ibm operations analytics predictive insights › itom › wp-content › uploads › sites › 39...
TRANSCRIPT
Operations Analytics Predictive Insights - Mediation Pack for AppDynamics Dynatrace DCRum Aternity and Splunk
IBM Operations Analytics Predictive Insights
Mediation Pack Group 2
Version 27
Document Revision 20190612
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
Edition notice
This edition applies to IBM Operations Analytics Predictive Insights 136 and to all subsequent releases and
modifications until otherwise indicated in new editions Please refer to the license files in the delivered
mediation pack for terms of use Also reference the IBM Operations Analytics Technical Community
(httpsdeveloperibmcomitoa) for additional information and forums on Predictive Insights
This mediation pack was provided in cooperation with Streamweaver Inc
See httpwwwstreamweavercomsolutionsopstream for additional information
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
iii
CONTENTS
Table of Contents
MEDIATION PACK GROUP 2 I
CONTENTS III
OVERVIEW 7
PREREQUISITE INFORMATION 9
SWP MEDIATORS OVERVIEW 11
KEY STEPS TO SETTING UP MEDIATORS 11
INSTALLING AND CONFIGURING 12
ZIP FILE CONTENTS 12 INSTALL PROCEDURE 12 NEW INSTALLATION 12 NOTES 13 STARTING SWP SYSTEM 15 STOPPING SWP SYSTEM 15 UPGRADE INSTALLATION 16 DEPLOYED DIRECTORY CONTENTS 16 ACCESSING THE MEDIATION SYSTEM 18
APPDYNAMICS MEDIATOR 20
OVERVIEW 20 MEDIATOR CREATION AND CONFIGURATION 20 SELECT SOURCE 20 CONNECT TO SOURCE 20 SELECT DESTINATION 21 CONNECT TO DESTINATION 21 SELECT DATA TYPES 21 CONFIGURING COLLECTORS 22 CONFIGURING DISTRIBUTORS 23 SAVE STREAM 23
AWS CLOUDWATCH MEDIATOR 24
OVERVIEW 24 EXTRACTION APPROACH 24 PRE-REQUISITES 24 MEDIATOR CREATION AND CONFIGURATION 26 CREATE NEW SOURCE 26
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
iv
CONNECT TO SOURCE 26 SELECT DESTINATION 27 CONNECT TO DESTINATION 27 SELECT DATA TYPES 27 CONFIGURING COLLECTORS 27 CONFIGURE DISTRIBUTORS 29 CONFIGURE STREAM 29
CA APM (INTROSCOPE) MEDIATOR 30
OVERVIEW 30 NOTE 30 MEDIATOR CREATION AND CONFIGURATION 30 SELECT SOURCE 30 CONNECT TO SOURCE 31 SELECT DESTINATION 32 CONNECT TO DESTINATION 32 SELECT DATA TYPES 32 CONFIGURING COLLECTORS 33 CONFIGURING DISTRIBUTORS 35 CONFIGURE STREAM 35
DYNATRACE MANAGED 36
OVERVIEW 36 EXTRACTION APPROACH 36 PRE-REQUISITES 36 MEDIATOR CREATION AND CONFIGURATION 36 CREATE NEW ORIGIN 36 CONNECT TO SOURCE 36 SELECT DESTINATION 37 CONNECT TO DESTINATION 37 SELECT DATA TYPES 38 CONFIGURING COLLECTORS 38 CONFIGURE DISTRIBUTORS 39 CONFIGURE STREAM 40
MF NNMI MEDIATOR 41
OVERVIEW 41 EXTRACTION APPROACH 41 PRE-REQUISITES 41 MEDIATOR CREATION AND CONFIGURATION 43 CREATE NEW SOURCE 43 CONNECT TO SOURCE 43 SELECT DESTINATION 44 CONNECT TO DESTINATION 44 SELECT DATA TYPES 44 CONFIGURING COLLECTORS 45
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
v
CONFIGURE DISTRIBUTORS 46 CONFIGURE STREAM 47
ORACLE OEM MEDIATOR 48
OVERVIEW 48 MEDIATOR CREATION AND CONFIGURATION 48 SELECT SOURCE 48 CONNECT TO SOURCE 48 SELECT DESTINATION 49 CONNECT TO DESTINATION 49 SELECT DATA TYPES 50 CONFIGURING COLLECTORS 50 CONFIGURING DISTRIBUTORS 51 SAVE STREAM 51
PIVOTAL CLOUD FOUNDRY MEDIATOR 52
OVERVIEW 52 EXTRACTION APPROACH 52 PRE-REQUISITES 52 OVERALL SETUP 53 MEDIATOR CREATION AND CONFIGURATION 54 CREATE NEW SOURCE 54 CONNECT TO SOURCE 54 SELECT DESTINATION 55 CONNECT TO DESTINATION 56 SELECT DATA TYPES 56 CONFIGURING COLLECTORS 56 CONFIGURE DISTRIBUTORS 58 CONFIGURE STREAM 59 NOZZLE SETUP 61 OPERATION 61
SAP HANA MEDIATOR 63
OVERVIEW 63 EXTRACTION APPROACH 63 SAP HANA PORT DEFAULTS 63 PRE-REQUISITES 63 MEDIATOR CREATION AND CONFIGURATION 63 CREATE NEW SOURCE 63 CONNECT TO SOURCE 64 SELECT DESTINATION 65 CONNECT TO DESTINATION 65 SELECT DATA TYPES 65 CONFIGURING COLLECTORS 66 CONFIGURE DISTRIBUTORS 67 CONFIGURE STREAM 68
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
vi
OPERATIONS 70
STARTING METRIC EXTRACTION 70 STOPPING METRIC EXTRACTION 70 RUNNING SINGLE EXTRACTIONS 70 EDITING AN EXISTING STREAM 72
PREDICTIVE INSIGHTS CSV OUTPUT 74
FILE DETAILS 74 TIMESTAMPS 76
PREDICTIVE INSIGHTS lsquoMODELrsquo FILES 77
GROUP INFORMATION 77 OBTAINING METRIC NAMES FOR PI MEDIATION TOOL 78
PREDICTIVE INSIGHTS REST OUTPUT 80
LOGS 81 CHANGING LOG LEVELS 81
SUPPORT 83
APPENDIX 1 SAP HANA METRICS 84
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
7
Overview
IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a
system through KPI metric data streams This includes data from various sources including
metrics gathered by industry performance and network monitoring solutions A mediation pack
helps users quickly integrate the KPI data streams by providing a default model to integrate a
particular performance manager
IBM Operations Analytics Predictive Insights Mediation Pack Group 2 known in the rest of this
document as the lsquoMediation Packrsquo provides a pre-built model as a starting point to facilitate data
ingestion Predictive Insights creates a performance model that it uses to detect or forecast
behavior outside the modeled range including generating an alarm when an anomalous behavior
occurs Doing so enables organizations to pro-actively avoid system and solution outages before
users are impacted
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
8
The Mediation Pack provides the capability to extract performance data and metrics from the
following sources
bull SAP Hana DB
bull CA APM (Introscope)
bull MicroFocus NNMi
bull AWS Cloudwatch
bull Pivotal Cloud Foundry
bull Dynatrace Managed
bull AppDynamics
bull Oracle OEM
and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This
document will cover the installation and setup of Mediation Pack for each of the supported source-
data technologies
The figure shows an overall view of the mediation solution in context
Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files
The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively
any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST
interface
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
9
Prerequisite Information
Item Description
Name Mediation Pack Group 2
Predictive Insights supported
versions
v136 and higher Note IBM PI continues to support the CSV
format for ingestion and so earlier versions of PI ie 133 will
also work)
Predictive Insights Information IBM Operations Analytics (demo trial resources)
httpswwwibmcomus-enmarketplaceit-operations-
analytics
IBM Operations Analytics Community (docs videos
mediation packs) httpsdeveloperibmcomitoa
IBM Operations Analytics Predictive Insights
documentation
httpswwwibmcomsupportknowledgecenterSSJQQ3
Solution Name supported
versions
Min version Max Version
CA APM (Introscope) 9x 107
SAP Hana 10 10
AWS Cloudwatch EC2 REST
API 2016-11-
15
Cloudwatch
111
Same as Min
version
MF NNMi 1040x 1040x
Pivotal Cloud Foundry 233 233
Dynatrace Managed Dec 2018 Jun 2019
AppDynamics 41x 459
Oracle OEM 13 13
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
10
Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack
Group 2
Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip
Supported Operating Systems Linux
RHEL 6x 7x 3264bit
CentOS 6x7x 3264bit
Java Java18+ is required There is a version included in the release
package Note Mediators do NOT run on IBM JVM (the one
delivered with PI at time of writing)
Web Browsers A modern HTML5CSS3Javascript browser Versions of
Chrome Firefox or Safari since 2012 should work Previous
browser versions may not work well or at all
Note Some versions of Internet Explorer (ie 9 or earlier)
have exhibited issues Please consider a more modern
browser to ensure full functionality of the product
Ports Port 7457 (default but configurable) is required by
Mediators
Outbound ports per Mediator Type
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
11
SWP Mediators Overview
The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)
specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are
support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files
2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface
Whilst each Mediator requires configuration specific to its source all the data flows through to a
corresponding Predictive Insights distributor The configurations approach for each of the
sources will be described in this document
The Mediators as part of the Mediation pack usually will be installed on the same host as
Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive
Insights If the Mediation pack is installed on a different host then mechanisms must be put in
place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS
FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond
the scope of this document In the case of REST distribution SWP does not have to be co-
located with Predictive Insights
Key steps to setting up Mediators
bull Installation
bull Configure one or more streams which extract the data from the supported sources
bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing
between CSV and REST output
bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This
should result in Predictive Insights CSV files being produced or JSON data
being posted to the IBM PI REST interface
bull Note Setting upconfiguring IBM PI REST interface is beyond the
scope of this document For details on this please refer to the
corresponding IBM PI documentation
bull In the case of CSV integration configure Predictive Insights for ingestion
using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights
lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not
used for PI REST integration
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
12
Installing and Configuring
Zip file contents
The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following
bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral
bull docs containing the User Guides
o for lsquostreamweaverrsquo (this document)
o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt
bull license containing license files
bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for
o Dynatrace
o DCRum
o Aternity
o Splunk
o ExtraHop
o CA UIM
o MF APM
o NetIQ AppManager
o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version
Install procedure
New Installation
The following section provides information for installing and configuring the Mediation Pack on
your IBM Operations Analytics Predictive Insights system
Normal lsquorpmrsquo conventions are followed
1 Verify Hardware amp Software Requirements
2 Verify user account privilege requirements
i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt
with root privileges This is required to create the Linux service
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
13
ii The installer will install as scadmindb2iadm (to match PI)
3 unzip mediation pack file
unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip
4 Install the bundled JVM
rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm
5 Install SWP Mediators Locate the rpm file in the expanded zip directory
rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-
1noarchrpm
After this you should see an indication of successful installation eg
SWP Mediators are installed by default to
optIBMstreamweaver-ibmpi
6 If you used root account exit and return to lsquoscadminrsquo user
Notes
Installing to an alternative location
You can install to a location other than optIBM as follows
sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-
ibmpi-27ibmpiltbuildgt-1noarchrpm
Running as non-`scadmin` user
As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships
If you wish to run as a non-`scadmin` user
Change ownership of all the files from down eg
chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi
Edit the initd startup script
etcinitdstreamweaver-api
Locate and edit the line specifying the DAEMON_USER
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin
Change the default `scadmin` to be your desired user eg
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
14
Co-existence of SWP-based and Accelerator-based mediators
If you are using the earlier Accelerator-based version along with this SWP-based version
note that they use the same default port setting of 7457 In this situation you should change the
default for SWP to something else You can do this by locating and editing the installed config
file in
optIBMstreamweaver-ibmpi confswpconf
and locate uncomment and change the line with port = 7457 to some other port
Settings for the Core Services services
Settings for the UI Service
ui
How the UI Services binds to a socket
connection
Hostname or IP Address to listen on The default
0000 causes
the REST API to listen on all interfaces
host = 0000
The port number to listen on
port = 7457
to some other free port
Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file
httpport = ltrest-port-numbergt must be one less than UI port this defines the
port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-
number server-side clients of REST service use this to locate it NOTE quotes are important here
Data directories
If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
Edition notice
This edition applies to IBM Operations Analytics Predictive Insights 136 and to all subsequent releases and
modifications until otherwise indicated in new editions Please refer to the license files in the delivered
mediation pack for terms of use Also reference the IBM Operations Analytics Technical Community
(httpsdeveloperibmcomitoa) for additional information and forums on Predictive Insights
This mediation pack was provided in cooperation with Streamweaver Inc
See httpwwwstreamweavercomsolutionsopstream for additional information
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
iii
CONTENTS
Table of Contents
MEDIATION PACK GROUP 2 I
CONTENTS III
OVERVIEW 7
PREREQUISITE INFORMATION 9
SWP MEDIATORS OVERVIEW 11
KEY STEPS TO SETTING UP MEDIATORS 11
INSTALLING AND CONFIGURING 12
ZIP FILE CONTENTS 12 INSTALL PROCEDURE 12 NEW INSTALLATION 12 NOTES 13 STARTING SWP SYSTEM 15 STOPPING SWP SYSTEM 15 UPGRADE INSTALLATION 16 DEPLOYED DIRECTORY CONTENTS 16 ACCESSING THE MEDIATION SYSTEM 18
APPDYNAMICS MEDIATOR 20
OVERVIEW 20 MEDIATOR CREATION AND CONFIGURATION 20 SELECT SOURCE 20 CONNECT TO SOURCE 20 SELECT DESTINATION 21 CONNECT TO DESTINATION 21 SELECT DATA TYPES 21 CONFIGURING COLLECTORS 22 CONFIGURING DISTRIBUTORS 23 SAVE STREAM 23
AWS CLOUDWATCH MEDIATOR 24
OVERVIEW 24 EXTRACTION APPROACH 24 PRE-REQUISITES 24 MEDIATOR CREATION AND CONFIGURATION 26 CREATE NEW SOURCE 26
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
iv
CONNECT TO SOURCE 26 SELECT DESTINATION 27 CONNECT TO DESTINATION 27 SELECT DATA TYPES 27 CONFIGURING COLLECTORS 27 CONFIGURE DISTRIBUTORS 29 CONFIGURE STREAM 29
CA APM (INTROSCOPE) MEDIATOR 30
OVERVIEW 30 NOTE 30 MEDIATOR CREATION AND CONFIGURATION 30 SELECT SOURCE 30 CONNECT TO SOURCE 31 SELECT DESTINATION 32 CONNECT TO DESTINATION 32 SELECT DATA TYPES 32 CONFIGURING COLLECTORS 33 CONFIGURING DISTRIBUTORS 35 CONFIGURE STREAM 35
DYNATRACE MANAGED 36
OVERVIEW 36 EXTRACTION APPROACH 36 PRE-REQUISITES 36 MEDIATOR CREATION AND CONFIGURATION 36 CREATE NEW ORIGIN 36 CONNECT TO SOURCE 36 SELECT DESTINATION 37 CONNECT TO DESTINATION 37 SELECT DATA TYPES 38 CONFIGURING COLLECTORS 38 CONFIGURE DISTRIBUTORS 39 CONFIGURE STREAM 40
MF NNMI MEDIATOR 41
OVERVIEW 41 EXTRACTION APPROACH 41 PRE-REQUISITES 41 MEDIATOR CREATION AND CONFIGURATION 43 CREATE NEW SOURCE 43 CONNECT TO SOURCE 43 SELECT DESTINATION 44 CONNECT TO DESTINATION 44 SELECT DATA TYPES 44 CONFIGURING COLLECTORS 45
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
v
CONFIGURE DISTRIBUTORS 46 CONFIGURE STREAM 47
ORACLE OEM MEDIATOR 48
OVERVIEW 48 MEDIATOR CREATION AND CONFIGURATION 48 SELECT SOURCE 48 CONNECT TO SOURCE 48 SELECT DESTINATION 49 CONNECT TO DESTINATION 49 SELECT DATA TYPES 50 CONFIGURING COLLECTORS 50 CONFIGURING DISTRIBUTORS 51 SAVE STREAM 51
PIVOTAL CLOUD FOUNDRY MEDIATOR 52
OVERVIEW 52 EXTRACTION APPROACH 52 PRE-REQUISITES 52 OVERALL SETUP 53 MEDIATOR CREATION AND CONFIGURATION 54 CREATE NEW SOURCE 54 CONNECT TO SOURCE 54 SELECT DESTINATION 55 CONNECT TO DESTINATION 56 SELECT DATA TYPES 56 CONFIGURING COLLECTORS 56 CONFIGURE DISTRIBUTORS 58 CONFIGURE STREAM 59 NOZZLE SETUP 61 OPERATION 61
SAP HANA MEDIATOR 63
OVERVIEW 63 EXTRACTION APPROACH 63 SAP HANA PORT DEFAULTS 63 PRE-REQUISITES 63 MEDIATOR CREATION AND CONFIGURATION 63 CREATE NEW SOURCE 63 CONNECT TO SOURCE 64 SELECT DESTINATION 65 CONNECT TO DESTINATION 65 SELECT DATA TYPES 65 CONFIGURING COLLECTORS 66 CONFIGURE DISTRIBUTORS 67 CONFIGURE STREAM 68
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
vi
OPERATIONS 70
STARTING METRIC EXTRACTION 70 STOPPING METRIC EXTRACTION 70 RUNNING SINGLE EXTRACTIONS 70 EDITING AN EXISTING STREAM 72
PREDICTIVE INSIGHTS CSV OUTPUT 74
FILE DETAILS 74 TIMESTAMPS 76
PREDICTIVE INSIGHTS lsquoMODELrsquo FILES 77
GROUP INFORMATION 77 OBTAINING METRIC NAMES FOR PI MEDIATION TOOL 78
PREDICTIVE INSIGHTS REST OUTPUT 80
LOGS 81 CHANGING LOG LEVELS 81
SUPPORT 83
APPENDIX 1 SAP HANA METRICS 84
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
7
Overview
IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a
system through KPI metric data streams This includes data from various sources including
metrics gathered by industry performance and network monitoring solutions A mediation pack
helps users quickly integrate the KPI data streams by providing a default model to integrate a
particular performance manager
IBM Operations Analytics Predictive Insights Mediation Pack Group 2 known in the rest of this
document as the lsquoMediation Packrsquo provides a pre-built model as a starting point to facilitate data
ingestion Predictive Insights creates a performance model that it uses to detect or forecast
behavior outside the modeled range including generating an alarm when an anomalous behavior
occurs Doing so enables organizations to pro-actively avoid system and solution outages before
users are impacted
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
8
The Mediation Pack provides the capability to extract performance data and metrics from the
following sources
bull SAP Hana DB
bull CA APM (Introscope)
bull MicroFocus NNMi
bull AWS Cloudwatch
bull Pivotal Cloud Foundry
bull Dynatrace Managed
bull AppDynamics
bull Oracle OEM
and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This
document will cover the installation and setup of Mediation Pack for each of the supported source-
data technologies
The figure shows an overall view of the mediation solution in context
Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files
The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively
any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST
interface
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
9
Prerequisite Information
Item Description
Name Mediation Pack Group 2
Predictive Insights supported
versions
v136 and higher Note IBM PI continues to support the CSV
format for ingestion and so earlier versions of PI ie 133 will
also work)
Predictive Insights Information IBM Operations Analytics (demo trial resources)
httpswwwibmcomus-enmarketplaceit-operations-
analytics
IBM Operations Analytics Community (docs videos
mediation packs) httpsdeveloperibmcomitoa
IBM Operations Analytics Predictive Insights
documentation
httpswwwibmcomsupportknowledgecenterSSJQQ3
Solution Name supported
versions
Min version Max Version
CA APM (Introscope) 9x 107
SAP Hana 10 10
AWS Cloudwatch EC2 REST
API 2016-11-
15
Cloudwatch
111
Same as Min
version
MF NNMi 1040x 1040x
Pivotal Cloud Foundry 233 233
Dynatrace Managed Dec 2018 Jun 2019
AppDynamics 41x 459
Oracle OEM 13 13
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
10
Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack
Group 2
Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip
Supported Operating Systems Linux
RHEL 6x 7x 3264bit
CentOS 6x7x 3264bit
Java Java18+ is required There is a version included in the release
package Note Mediators do NOT run on IBM JVM (the one
delivered with PI at time of writing)
Web Browsers A modern HTML5CSS3Javascript browser Versions of
Chrome Firefox or Safari since 2012 should work Previous
browser versions may not work well or at all
Note Some versions of Internet Explorer (ie 9 or earlier)
have exhibited issues Please consider a more modern
browser to ensure full functionality of the product
Ports Port 7457 (default but configurable) is required by
Mediators
Outbound ports per Mediator Type
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
11
SWP Mediators Overview
The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)
specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are
support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files
2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface
Whilst each Mediator requires configuration specific to its source all the data flows through to a
corresponding Predictive Insights distributor The configurations approach for each of the
sources will be described in this document
The Mediators as part of the Mediation pack usually will be installed on the same host as
Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive
Insights If the Mediation pack is installed on a different host then mechanisms must be put in
place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS
FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond
the scope of this document In the case of REST distribution SWP does not have to be co-
located with Predictive Insights
Key steps to setting up Mediators
bull Installation
bull Configure one or more streams which extract the data from the supported sources
bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing
between CSV and REST output
bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This
should result in Predictive Insights CSV files being produced or JSON data
being posted to the IBM PI REST interface
bull Note Setting upconfiguring IBM PI REST interface is beyond the
scope of this document For details on this please refer to the
corresponding IBM PI documentation
bull In the case of CSV integration configure Predictive Insights for ingestion
using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights
lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not
used for PI REST integration
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
12
Installing and Configuring
Zip file contents
The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following
bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral
bull docs containing the User Guides
o for lsquostreamweaverrsquo (this document)
o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt
bull license containing license files
bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for
o Dynatrace
o DCRum
o Aternity
o Splunk
o ExtraHop
o CA UIM
o MF APM
o NetIQ AppManager
o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version
Install procedure
New Installation
The following section provides information for installing and configuring the Mediation Pack on
your IBM Operations Analytics Predictive Insights system
Normal lsquorpmrsquo conventions are followed
1 Verify Hardware amp Software Requirements
2 Verify user account privilege requirements
i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt
with root privileges This is required to create the Linux service
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
13
ii The installer will install as scadmindb2iadm (to match PI)
3 unzip mediation pack file
unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip
4 Install the bundled JVM
rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm
5 Install SWP Mediators Locate the rpm file in the expanded zip directory
rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-
1noarchrpm
After this you should see an indication of successful installation eg
SWP Mediators are installed by default to
optIBMstreamweaver-ibmpi
6 If you used root account exit and return to lsquoscadminrsquo user
Notes
Installing to an alternative location
You can install to a location other than optIBM as follows
sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-
ibmpi-27ibmpiltbuildgt-1noarchrpm
Running as non-`scadmin` user
As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships
If you wish to run as a non-`scadmin` user
Change ownership of all the files from down eg
chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi
Edit the initd startup script
etcinitdstreamweaver-api
Locate and edit the line specifying the DAEMON_USER
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin
Change the default `scadmin` to be your desired user eg
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
14
Co-existence of SWP-based and Accelerator-based mediators
If you are using the earlier Accelerator-based version along with this SWP-based version
note that they use the same default port setting of 7457 In this situation you should change the
default for SWP to something else You can do this by locating and editing the installed config
file in
optIBMstreamweaver-ibmpi confswpconf
and locate uncomment and change the line with port = 7457 to some other port
Settings for the Core Services services
Settings for the UI Service
ui
How the UI Services binds to a socket
connection
Hostname or IP Address to listen on The default
0000 causes
the REST API to listen on all interfaces
host = 0000
The port number to listen on
port = 7457
to some other free port
Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file
httpport = ltrest-port-numbergt must be one less than UI port this defines the
port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-
number server-side clients of REST service use this to locate it NOTE quotes are important here
Data directories
If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
iii
CONTENTS
Table of Contents
MEDIATION PACK GROUP 2 I
CONTENTS III
OVERVIEW 7
PREREQUISITE INFORMATION 9
SWP MEDIATORS OVERVIEW 11
KEY STEPS TO SETTING UP MEDIATORS 11
INSTALLING AND CONFIGURING 12
ZIP FILE CONTENTS 12 INSTALL PROCEDURE 12 NEW INSTALLATION 12 NOTES 13 STARTING SWP SYSTEM 15 STOPPING SWP SYSTEM 15 UPGRADE INSTALLATION 16 DEPLOYED DIRECTORY CONTENTS 16 ACCESSING THE MEDIATION SYSTEM 18
APPDYNAMICS MEDIATOR 20
OVERVIEW 20 MEDIATOR CREATION AND CONFIGURATION 20 SELECT SOURCE 20 CONNECT TO SOURCE 20 SELECT DESTINATION 21 CONNECT TO DESTINATION 21 SELECT DATA TYPES 21 CONFIGURING COLLECTORS 22 CONFIGURING DISTRIBUTORS 23 SAVE STREAM 23
AWS CLOUDWATCH MEDIATOR 24
OVERVIEW 24 EXTRACTION APPROACH 24 PRE-REQUISITES 24 MEDIATOR CREATION AND CONFIGURATION 26 CREATE NEW SOURCE 26
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
iv
CONNECT TO SOURCE 26 SELECT DESTINATION 27 CONNECT TO DESTINATION 27 SELECT DATA TYPES 27 CONFIGURING COLLECTORS 27 CONFIGURE DISTRIBUTORS 29 CONFIGURE STREAM 29
CA APM (INTROSCOPE) MEDIATOR 30
OVERVIEW 30 NOTE 30 MEDIATOR CREATION AND CONFIGURATION 30 SELECT SOURCE 30 CONNECT TO SOURCE 31 SELECT DESTINATION 32 CONNECT TO DESTINATION 32 SELECT DATA TYPES 32 CONFIGURING COLLECTORS 33 CONFIGURING DISTRIBUTORS 35 CONFIGURE STREAM 35
DYNATRACE MANAGED 36
OVERVIEW 36 EXTRACTION APPROACH 36 PRE-REQUISITES 36 MEDIATOR CREATION AND CONFIGURATION 36 CREATE NEW ORIGIN 36 CONNECT TO SOURCE 36 SELECT DESTINATION 37 CONNECT TO DESTINATION 37 SELECT DATA TYPES 38 CONFIGURING COLLECTORS 38 CONFIGURE DISTRIBUTORS 39 CONFIGURE STREAM 40
MF NNMI MEDIATOR 41
OVERVIEW 41 EXTRACTION APPROACH 41 PRE-REQUISITES 41 MEDIATOR CREATION AND CONFIGURATION 43 CREATE NEW SOURCE 43 CONNECT TO SOURCE 43 SELECT DESTINATION 44 CONNECT TO DESTINATION 44 SELECT DATA TYPES 44 CONFIGURING COLLECTORS 45
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
v
CONFIGURE DISTRIBUTORS 46 CONFIGURE STREAM 47
ORACLE OEM MEDIATOR 48
OVERVIEW 48 MEDIATOR CREATION AND CONFIGURATION 48 SELECT SOURCE 48 CONNECT TO SOURCE 48 SELECT DESTINATION 49 CONNECT TO DESTINATION 49 SELECT DATA TYPES 50 CONFIGURING COLLECTORS 50 CONFIGURING DISTRIBUTORS 51 SAVE STREAM 51
PIVOTAL CLOUD FOUNDRY MEDIATOR 52
OVERVIEW 52 EXTRACTION APPROACH 52 PRE-REQUISITES 52 OVERALL SETUP 53 MEDIATOR CREATION AND CONFIGURATION 54 CREATE NEW SOURCE 54 CONNECT TO SOURCE 54 SELECT DESTINATION 55 CONNECT TO DESTINATION 56 SELECT DATA TYPES 56 CONFIGURING COLLECTORS 56 CONFIGURE DISTRIBUTORS 58 CONFIGURE STREAM 59 NOZZLE SETUP 61 OPERATION 61
SAP HANA MEDIATOR 63
OVERVIEW 63 EXTRACTION APPROACH 63 SAP HANA PORT DEFAULTS 63 PRE-REQUISITES 63 MEDIATOR CREATION AND CONFIGURATION 63 CREATE NEW SOURCE 63 CONNECT TO SOURCE 64 SELECT DESTINATION 65 CONNECT TO DESTINATION 65 SELECT DATA TYPES 65 CONFIGURING COLLECTORS 66 CONFIGURE DISTRIBUTORS 67 CONFIGURE STREAM 68
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
vi
OPERATIONS 70
STARTING METRIC EXTRACTION 70 STOPPING METRIC EXTRACTION 70 RUNNING SINGLE EXTRACTIONS 70 EDITING AN EXISTING STREAM 72
PREDICTIVE INSIGHTS CSV OUTPUT 74
FILE DETAILS 74 TIMESTAMPS 76
PREDICTIVE INSIGHTS lsquoMODELrsquo FILES 77
GROUP INFORMATION 77 OBTAINING METRIC NAMES FOR PI MEDIATION TOOL 78
PREDICTIVE INSIGHTS REST OUTPUT 80
LOGS 81 CHANGING LOG LEVELS 81
SUPPORT 83
APPENDIX 1 SAP HANA METRICS 84
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
7
Overview
IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a
system through KPI metric data streams This includes data from various sources including
metrics gathered by industry performance and network monitoring solutions A mediation pack
helps users quickly integrate the KPI data streams by providing a default model to integrate a
particular performance manager
IBM Operations Analytics Predictive Insights Mediation Pack Group 2 known in the rest of this
document as the lsquoMediation Packrsquo provides a pre-built model as a starting point to facilitate data
ingestion Predictive Insights creates a performance model that it uses to detect or forecast
behavior outside the modeled range including generating an alarm when an anomalous behavior
occurs Doing so enables organizations to pro-actively avoid system and solution outages before
users are impacted
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
8
The Mediation Pack provides the capability to extract performance data and metrics from the
following sources
bull SAP Hana DB
bull CA APM (Introscope)
bull MicroFocus NNMi
bull AWS Cloudwatch
bull Pivotal Cloud Foundry
bull Dynatrace Managed
bull AppDynamics
bull Oracle OEM
and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This
document will cover the installation and setup of Mediation Pack for each of the supported source-
data technologies
The figure shows an overall view of the mediation solution in context
Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files
The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively
any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST
interface
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
9
Prerequisite Information
Item Description
Name Mediation Pack Group 2
Predictive Insights supported
versions
v136 and higher Note IBM PI continues to support the CSV
format for ingestion and so earlier versions of PI ie 133 will
also work)
Predictive Insights Information IBM Operations Analytics (demo trial resources)
httpswwwibmcomus-enmarketplaceit-operations-
analytics
IBM Operations Analytics Community (docs videos
mediation packs) httpsdeveloperibmcomitoa
IBM Operations Analytics Predictive Insights
documentation
httpswwwibmcomsupportknowledgecenterSSJQQ3
Solution Name supported
versions
Min version Max Version
CA APM (Introscope) 9x 107
SAP Hana 10 10
AWS Cloudwatch EC2 REST
API 2016-11-
15
Cloudwatch
111
Same as Min
version
MF NNMi 1040x 1040x
Pivotal Cloud Foundry 233 233
Dynatrace Managed Dec 2018 Jun 2019
AppDynamics 41x 459
Oracle OEM 13 13
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
10
Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack
Group 2
Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip
Supported Operating Systems Linux
RHEL 6x 7x 3264bit
CentOS 6x7x 3264bit
Java Java18+ is required There is a version included in the release
package Note Mediators do NOT run on IBM JVM (the one
delivered with PI at time of writing)
Web Browsers A modern HTML5CSS3Javascript browser Versions of
Chrome Firefox or Safari since 2012 should work Previous
browser versions may not work well or at all
Note Some versions of Internet Explorer (ie 9 or earlier)
have exhibited issues Please consider a more modern
browser to ensure full functionality of the product
Ports Port 7457 (default but configurable) is required by
Mediators
Outbound ports per Mediator Type
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
11
SWP Mediators Overview
The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)
specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are
support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files
2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface
Whilst each Mediator requires configuration specific to its source all the data flows through to a
corresponding Predictive Insights distributor The configurations approach for each of the
sources will be described in this document
The Mediators as part of the Mediation pack usually will be installed on the same host as
Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive
Insights If the Mediation pack is installed on a different host then mechanisms must be put in
place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS
FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond
the scope of this document In the case of REST distribution SWP does not have to be co-
located with Predictive Insights
Key steps to setting up Mediators
bull Installation
bull Configure one or more streams which extract the data from the supported sources
bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing
between CSV and REST output
bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This
should result in Predictive Insights CSV files being produced or JSON data
being posted to the IBM PI REST interface
bull Note Setting upconfiguring IBM PI REST interface is beyond the
scope of this document For details on this please refer to the
corresponding IBM PI documentation
bull In the case of CSV integration configure Predictive Insights for ingestion
using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights
lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not
used for PI REST integration
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
12
Installing and Configuring
Zip file contents
The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following
bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral
bull docs containing the User Guides
o for lsquostreamweaverrsquo (this document)
o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt
bull license containing license files
bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for
o Dynatrace
o DCRum
o Aternity
o Splunk
o ExtraHop
o CA UIM
o MF APM
o NetIQ AppManager
o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version
Install procedure
New Installation
The following section provides information for installing and configuring the Mediation Pack on
your IBM Operations Analytics Predictive Insights system
Normal lsquorpmrsquo conventions are followed
1 Verify Hardware amp Software Requirements
2 Verify user account privilege requirements
i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt
with root privileges This is required to create the Linux service
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
13
ii The installer will install as scadmindb2iadm (to match PI)
3 unzip mediation pack file
unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip
4 Install the bundled JVM
rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm
5 Install SWP Mediators Locate the rpm file in the expanded zip directory
rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-
1noarchrpm
After this you should see an indication of successful installation eg
SWP Mediators are installed by default to
optIBMstreamweaver-ibmpi
6 If you used root account exit and return to lsquoscadminrsquo user
Notes
Installing to an alternative location
You can install to a location other than optIBM as follows
sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-
ibmpi-27ibmpiltbuildgt-1noarchrpm
Running as non-`scadmin` user
As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships
If you wish to run as a non-`scadmin` user
Change ownership of all the files from down eg
chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi
Edit the initd startup script
etcinitdstreamweaver-api
Locate and edit the line specifying the DAEMON_USER
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin
Change the default `scadmin` to be your desired user eg
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
14
Co-existence of SWP-based and Accelerator-based mediators
If you are using the earlier Accelerator-based version along with this SWP-based version
note that they use the same default port setting of 7457 In this situation you should change the
default for SWP to something else You can do this by locating and editing the installed config
file in
optIBMstreamweaver-ibmpi confswpconf
and locate uncomment and change the line with port = 7457 to some other port
Settings for the Core Services services
Settings for the UI Service
ui
How the UI Services binds to a socket
connection
Hostname or IP Address to listen on The default
0000 causes
the REST API to listen on all interfaces
host = 0000
The port number to listen on
port = 7457
to some other free port
Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file
httpport = ltrest-port-numbergt must be one less than UI port this defines the
port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-
number server-side clients of REST service use this to locate it NOTE quotes are important here
Data directories
If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
iv
CONNECT TO SOURCE 26 SELECT DESTINATION 27 CONNECT TO DESTINATION 27 SELECT DATA TYPES 27 CONFIGURING COLLECTORS 27 CONFIGURE DISTRIBUTORS 29 CONFIGURE STREAM 29
CA APM (INTROSCOPE) MEDIATOR 30
OVERVIEW 30 NOTE 30 MEDIATOR CREATION AND CONFIGURATION 30 SELECT SOURCE 30 CONNECT TO SOURCE 31 SELECT DESTINATION 32 CONNECT TO DESTINATION 32 SELECT DATA TYPES 32 CONFIGURING COLLECTORS 33 CONFIGURING DISTRIBUTORS 35 CONFIGURE STREAM 35
DYNATRACE MANAGED 36
OVERVIEW 36 EXTRACTION APPROACH 36 PRE-REQUISITES 36 MEDIATOR CREATION AND CONFIGURATION 36 CREATE NEW ORIGIN 36 CONNECT TO SOURCE 36 SELECT DESTINATION 37 CONNECT TO DESTINATION 37 SELECT DATA TYPES 38 CONFIGURING COLLECTORS 38 CONFIGURE DISTRIBUTORS 39 CONFIGURE STREAM 40
MF NNMI MEDIATOR 41
OVERVIEW 41 EXTRACTION APPROACH 41 PRE-REQUISITES 41 MEDIATOR CREATION AND CONFIGURATION 43 CREATE NEW SOURCE 43 CONNECT TO SOURCE 43 SELECT DESTINATION 44 CONNECT TO DESTINATION 44 SELECT DATA TYPES 44 CONFIGURING COLLECTORS 45
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
v
CONFIGURE DISTRIBUTORS 46 CONFIGURE STREAM 47
ORACLE OEM MEDIATOR 48
OVERVIEW 48 MEDIATOR CREATION AND CONFIGURATION 48 SELECT SOURCE 48 CONNECT TO SOURCE 48 SELECT DESTINATION 49 CONNECT TO DESTINATION 49 SELECT DATA TYPES 50 CONFIGURING COLLECTORS 50 CONFIGURING DISTRIBUTORS 51 SAVE STREAM 51
PIVOTAL CLOUD FOUNDRY MEDIATOR 52
OVERVIEW 52 EXTRACTION APPROACH 52 PRE-REQUISITES 52 OVERALL SETUP 53 MEDIATOR CREATION AND CONFIGURATION 54 CREATE NEW SOURCE 54 CONNECT TO SOURCE 54 SELECT DESTINATION 55 CONNECT TO DESTINATION 56 SELECT DATA TYPES 56 CONFIGURING COLLECTORS 56 CONFIGURE DISTRIBUTORS 58 CONFIGURE STREAM 59 NOZZLE SETUP 61 OPERATION 61
SAP HANA MEDIATOR 63
OVERVIEW 63 EXTRACTION APPROACH 63 SAP HANA PORT DEFAULTS 63 PRE-REQUISITES 63 MEDIATOR CREATION AND CONFIGURATION 63 CREATE NEW SOURCE 63 CONNECT TO SOURCE 64 SELECT DESTINATION 65 CONNECT TO DESTINATION 65 SELECT DATA TYPES 65 CONFIGURING COLLECTORS 66 CONFIGURE DISTRIBUTORS 67 CONFIGURE STREAM 68
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
vi
OPERATIONS 70
STARTING METRIC EXTRACTION 70 STOPPING METRIC EXTRACTION 70 RUNNING SINGLE EXTRACTIONS 70 EDITING AN EXISTING STREAM 72
PREDICTIVE INSIGHTS CSV OUTPUT 74
FILE DETAILS 74 TIMESTAMPS 76
PREDICTIVE INSIGHTS lsquoMODELrsquo FILES 77
GROUP INFORMATION 77 OBTAINING METRIC NAMES FOR PI MEDIATION TOOL 78
PREDICTIVE INSIGHTS REST OUTPUT 80
LOGS 81 CHANGING LOG LEVELS 81
SUPPORT 83
APPENDIX 1 SAP HANA METRICS 84
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
7
Overview
IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a
system through KPI metric data streams This includes data from various sources including
metrics gathered by industry performance and network monitoring solutions A mediation pack
helps users quickly integrate the KPI data streams by providing a default model to integrate a
particular performance manager
IBM Operations Analytics Predictive Insights Mediation Pack Group 2 known in the rest of this
document as the lsquoMediation Packrsquo provides a pre-built model as a starting point to facilitate data
ingestion Predictive Insights creates a performance model that it uses to detect or forecast
behavior outside the modeled range including generating an alarm when an anomalous behavior
occurs Doing so enables organizations to pro-actively avoid system and solution outages before
users are impacted
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
8
The Mediation Pack provides the capability to extract performance data and metrics from the
following sources
bull SAP Hana DB
bull CA APM (Introscope)
bull MicroFocus NNMi
bull AWS Cloudwatch
bull Pivotal Cloud Foundry
bull Dynatrace Managed
bull AppDynamics
bull Oracle OEM
and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This
document will cover the installation and setup of Mediation Pack for each of the supported source-
data technologies
The figure shows an overall view of the mediation solution in context
Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files
The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively
any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST
interface
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
9
Prerequisite Information
Item Description
Name Mediation Pack Group 2
Predictive Insights supported
versions
v136 and higher Note IBM PI continues to support the CSV
format for ingestion and so earlier versions of PI ie 133 will
also work)
Predictive Insights Information IBM Operations Analytics (demo trial resources)
httpswwwibmcomus-enmarketplaceit-operations-
analytics
IBM Operations Analytics Community (docs videos
mediation packs) httpsdeveloperibmcomitoa
IBM Operations Analytics Predictive Insights
documentation
httpswwwibmcomsupportknowledgecenterSSJQQ3
Solution Name supported
versions
Min version Max Version
CA APM (Introscope) 9x 107
SAP Hana 10 10
AWS Cloudwatch EC2 REST
API 2016-11-
15
Cloudwatch
111
Same as Min
version
MF NNMi 1040x 1040x
Pivotal Cloud Foundry 233 233
Dynatrace Managed Dec 2018 Jun 2019
AppDynamics 41x 459
Oracle OEM 13 13
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
10
Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack
Group 2
Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip
Supported Operating Systems Linux
RHEL 6x 7x 3264bit
CentOS 6x7x 3264bit
Java Java18+ is required There is a version included in the release
package Note Mediators do NOT run on IBM JVM (the one
delivered with PI at time of writing)
Web Browsers A modern HTML5CSS3Javascript browser Versions of
Chrome Firefox or Safari since 2012 should work Previous
browser versions may not work well or at all
Note Some versions of Internet Explorer (ie 9 or earlier)
have exhibited issues Please consider a more modern
browser to ensure full functionality of the product
Ports Port 7457 (default but configurable) is required by
Mediators
Outbound ports per Mediator Type
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
11
SWP Mediators Overview
The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)
specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are
support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files
2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface
Whilst each Mediator requires configuration specific to its source all the data flows through to a
corresponding Predictive Insights distributor The configurations approach for each of the
sources will be described in this document
The Mediators as part of the Mediation pack usually will be installed on the same host as
Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive
Insights If the Mediation pack is installed on a different host then mechanisms must be put in
place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS
FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond
the scope of this document In the case of REST distribution SWP does not have to be co-
located with Predictive Insights
Key steps to setting up Mediators
bull Installation
bull Configure one or more streams which extract the data from the supported sources
bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing
between CSV and REST output
bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This
should result in Predictive Insights CSV files being produced or JSON data
being posted to the IBM PI REST interface
bull Note Setting upconfiguring IBM PI REST interface is beyond the
scope of this document For details on this please refer to the
corresponding IBM PI documentation
bull In the case of CSV integration configure Predictive Insights for ingestion
using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights
lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not
used for PI REST integration
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
12
Installing and Configuring
Zip file contents
The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following
bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral
bull docs containing the User Guides
o for lsquostreamweaverrsquo (this document)
o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt
bull license containing license files
bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for
o Dynatrace
o DCRum
o Aternity
o Splunk
o ExtraHop
o CA UIM
o MF APM
o NetIQ AppManager
o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version
Install procedure
New Installation
The following section provides information for installing and configuring the Mediation Pack on
your IBM Operations Analytics Predictive Insights system
Normal lsquorpmrsquo conventions are followed
1 Verify Hardware amp Software Requirements
2 Verify user account privilege requirements
i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt
with root privileges This is required to create the Linux service
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
13
ii The installer will install as scadmindb2iadm (to match PI)
3 unzip mediation pack file
unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip
4 Install the bundled JVM
rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm
5 Install SWP Mediators Locate the rpm file in the expanded zip directory
rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-
1noarchrpm
After this you should see an indication of successful installation eg
SWP Mediators are installed by default to
optIBMstreamweaver-ibmpi
6 If you used root account exit and return to lsquoscadminrsquo user
Notes
Installing to an alternative location
You can install to a location other than optIBM as follows
sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-
ibmpi-27ibmpiltbuildgt-1noarchrpm
Running as non-`scadmin` user
As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships
If you wish to run as a non-`scadmin` user
Change ownership of all the files from down eg
chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi
Edit the initd startup script
etcinitdstreamweaver-api
Locate and edit the line specifying the DAEMON_USER
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin
Change the default `scadmin` to be your desired user eg
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
14
Co-existence of SWP-based and Accelerator-based mediators
If you are using the earlier Accelerator-based version along with this SWP-based version
note that they use the same default port setting of 7457 In this situation you should change the
default for SWP to something else You can do this by locating and editing the installed config
file in
optIBMstreamweaver-ibmpi confswpconf
and locate uncomment and change the line with port = 7457 to some other port
Settings for the Core Services services
Settings for the UI Service
ui
How the UI Services binds to a socket
connection
Hostname or IP Address to listen on The default
0000 causes
the REST API to listen on all interfaces
host = 0000
The port number to listen on
port = 7457
to some other free port
Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file
httpport = ltrest-port-numbergt must be one less than UI port this defines the
port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-
number server-side clients of REST service use this to locate it NOTE quotes are important here
Data directories
If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
v
CONFIGURE DISTRIBUTORS 46 CONFIGURE STREAM 47
ORACLE OEM MEDIATOR 48
OVERVIEW 48 MEDIATOR CREATION AND CONFIGURATION 48 SELECT SOURCE 48 CONNECT TO SOURCE 48 SELECT DESTINATION 49 CONNECT TO DESTINATION 49 SELECT DATA TYPES 50 CONFIGURING COLLECTORS 50 CONFIGURING DISTRIBUTORS 51 SAVE STREAM 51
PIVOTAL CLOUD FOUNDRY MEDIATOR 52
OVERVIEW 52 EXTRACTION APPROACH 52 PRE-REQUISITES 52 OVERALL SETUP 53 MEDIATOR CREATION AND CONFIGURATION 54 CREATE NEW SOURCE 54 CONNECT TO SOURCE 54 SELECT DESTINATION 55 CONNECT TO DESTINATION 56 SELECT DATA TYPES 56 CONFIGURING COLLECTORS 56 CONFIGURE DISTRIBUTORS 58 CONFIGURE STREAM 59 NOZZLE SETUP 61 OPERATION 61
SAP HANA MEDIATOR 63
OVERVIEW 63 EXTRACTION APPROACH 63 SAP HANA PORT DEFAULTS 63 PRE-REQUISITES 63 MEDIATOR CREATION AND CONFIGURATION 63 CREATE NEW SOURCE 63 CONNECT TO SOURCE 64 SELECT DESTINATION 65 CONNECT TO DESTINATION 65 SELECT DATA TYPES 65 CONFIGURING COLLECTORS 66 CONFIGURE DISTRIBUTORS 67 CONFIGURE STREAM 68
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
vi
OPERATIONS 70
STARTING METRIC EXTRACTION 70 STOPPING METRIC EXTRACTION 70 RUNNING SINGLE EXTRACTIONS 70 EDITING AN EXISTING STREAM 72
PREDICTIVE INSIGHTS CSV OUTPUT 74
FILE DETAILS 74 TIMESTAMPS 76
PREDICTIVE INSIGHTS lsquoMODELrsquo FILES 77
GROUP INFORMATION 77 OBTAINING METRIC NAMES FOR PI MEDIATION TOOL 78
PREDICTIVE INSIGHTS REST OUTPUT 80
LOGS 81 CHANGING LOG LEVELS 81
SUPPORT 83
APPENDIX 1 SAP HANA METRICS 84
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
7
Overview
IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a
system through KPI metric data streams This includes data from various sources including
metrics gathered by industry performance and network monitoring solutions A mediation pack
helps users quickly integrate the KPI data streams by providing a default model to integrate a
particular performance manager
IBM Operations Analytics Predictive Insights Mediation Pack Group 2 known in the rest of this
document as the lsquoMediation Packrsquo provides a pre-built model as a starting point to facilitate data
ingestion Predictive Insights creates a performance model that it uses to detect or forecast
behavior outside the modeled range including generating an alarm when an anomalous behavior
occurs Doing so enables organizations to pro-actively avoid system and solution outages before
users are impacted
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
8
The Mediation Pack provides the capability to extract performance data and metrics from the
following sources
bull SAP Hana DB
bull CA APM (Introscope)
bull MicroFocus NNMi
bull AWS Cloudwatch
bull Pivotal Cloud Foundry
bull Dynatrace Managed
bull AppDynamics
bull Oracle OEM
and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This
document will cover the installation and setup of Mediation Pack for each of the supported source-
data technologies
The figure shows an overall view of the mediation solution in context
Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files
The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively
any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST
interface
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
9
Prerequisite Information
Item Description
Name Mediation Pack Group 2
Predictive Insights supported
versions
v136 and higher Note IBM PI continues to support the CSV
format for ingestion and so earlier versions of PI ie 133 will
also work)
Predictive Insights Information IBM Operations Analytics (demo trial resources)
httpswwwibmcomus-enmarketplaceit-operations-
analytics
IBM Operations Analytics Community (docs videos
mediation packs) httpsdeveloperibmcomitoa
IBM Operations Analytics Predictive Insights
documentation
httpswwwibmcomsupportknowledgecenterSSJQQ3
Solution Name supported
versions
Min version Max Version
CA APM (Introscope) 9x 107
SAP Hana 10 10
AWS Cloudwatch EC2 REST
API 2016-11-
15
Cloudwatch
111
Same as Min
version
MF NNMi 1040x 1040x
Pivotal Cloud Foundry 233 233
Dynatrace Managed Dec 2018 Jun 2019
AppDynamics 41x 459
Oracle OEM 13 13
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
10
Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack
Group 2
Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip
Supported Operating Systems Linux
RHEL 6x 7x 3264bit
CentOS 6x7x 3264bit
Java Java18+ is required There is a version included in the release
package Note Mediators do NOT run on IBM JVM (the one
delivered with PI at time of writing)
Web Browsers A modern HTML5CSS3Javascript browser Versions of
Chrome Firefox or Safari since 2012 should work Previous
browser versions may not work well or at all
Note Some versions of Internet Explorer (ie 9 or earlier)
have exhibited issues Please consider a more modern
browser to ensure full functionality of the product
Ports Port 7457 (default but configurable) is required by
Mediators
Outbound ports per Mediator Type
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
11
SWP Mediators Overview
The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)
specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are
support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files
2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface
Whilst each Mediator requires configuration specific to its source all the data flows through to a
corresponding Predictive Insights distributor The configurations approach for each of the
sources will be described in this document
The Mediators as part of the Mediation pack usually will be installed on the same host as
Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive
Insights If the Mediation pack is installed on a different host then mechanisms must be put in
place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS
FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond
the scope of this document In the case of REST distribution SWP does not have to be co-
located with Predictive Insights
Key steps to setting up Mediators
bull Installation
bull Configure one or more streams which extract the data from the supported sources
bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing
between CSV and REST output
bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This
should result in Predictive Insights CSV files being produced or JSON data
being posted to the IBM PI REST interface
bull Note Setting upconfiguring IBM PI REST interface is beyond the
scope of this document For details on this please refer to the
corresponding IBM PI documentation
bull In the case of CSV integration configure Predictive Insights for ingestion
using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights
lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not
used for PI REST integration
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
12
Installing and Configuring
Zip file contents
The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following
bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral
bull docs containing the User Guides
o for lsquostreamweaverrsquo (this document)
o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt
bull license containing license files
bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for
o Dynatrace
o DCRum
o Aternity
o Splunk
o ExtraHop
o CA UIM
o MF APM
o NetIQ AppManager
o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version
Install procedure
New Installation
The following section provides information for installing and configuring the Mediation Pack on
your IBM Operations Analytics Predictive Insights system
Normal lsquorpmrsquo conventions are followed
1 Verify Hardware amp Software Requirements
2 Verify user account privilege requirements
i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt
with root privileges This is required to create the Linux service
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
13
ii The installer will install as scadmindb2iadm (to match PI)
3 unzip mediation pack file
unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip
4 Install the bundled JVM
rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm
5 Install SWP Mediators Locate the rpm file in the expanded zip directory
rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-
1noarchrpm
After this you should see an indication of successful installation eg
SWP Mediators are installed by default to
optIBMstreamweaver-ibmpi
6 If you used root account exit and return to lsquoscadminrsquo user
Notes
Installing to an alternative location
You can install to a location other than optIBM as follows
sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-
ibmpi-27ibmpiltbuildgt-1noarchrpm
Running as non-`scadmin` user
As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships
If you wish to run as a non-`scadmin` user
Change ownership of all the files from down eg
chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi
Edit the initd startup script
etcinitdstreamweaver-api
Locate and edit the line specifying the DAEMON_USER
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin
Change the default `scadmin` to be your desired user eg
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
14
Co-existence of SWP-based and Accelerator-based mediators
If you are using the earlier Accelerator-based version along with this SWP-based version
note that they use the same default port setting of 7457 In this situation you should change the
default for SWP to something else You can do this by locating and editing the installed config
file in
optIBMstreamweaver-ibmpi confswpconf
and locate uncomment and change the line with port = 7457 to some other port
Settings for the Core Services services
Settings for the UI Service
ui
How the UI Services binds to a socket
connection
Hostname or IP Address to listen on The default
0000 causes
the REST API to listen on all interfaces
host = 0000
The port number to listen on
port = 7457
to some other free port
Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file
httpport = ltrest-port-numbergt must be one less than UI port this defines the
port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-
number server-side clients of REST service use this to locate it NOTE quotes are important here
Data directories
If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
vi
OPERATIONS 70
STARTING METRIC EXTRACTION 70 STOPPING METRIC EXTRACTION 70 RUNNING SINGLE EXTRACTIONS 70 EDITING AN EXISTING STREAM 72
PREDICTIVE INSIGHTS CSV OUTPUT 74
FILE DETAILS 74 TIMESTAMPS 76
PREDICTIVE INSIGHTS lsquoMODELrsquo FILES 77
GROUP INFORMATION 77 OBTAINING METRIC NAMES FOR PI MEDIATION TOOL 78
PREDICTIVE INSIGHTS REST OUTPUT 80
LOGS 81 CHANGING LOG LEVELS 81
SUPPORT 83
APPENDIX 1 SAP HANA METRICS 84
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
7
Overview
IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a
system through KPI metric data streams This includes data from various sources including
metrics gathered by industry performance and network monitoring solutions A mediation pack
helps users quickly integrate the KPI data streams by providing a default model to integrate a
particular performance manager
IBM Operations Analytics Predictive Insights Mediation Pack Group 2 known in the rest of this
document as the lsquoMediation Packrsquo provides a pre-built model as a starting point to facilitate data
ingestion Predictive Insights creates a performance model that it uses to detect or forecast
behavior outside the modeled range including generating an alarm when an anomalous behavior
occurs Doing so enables organizations to pro-actively avoid system and solution outages before
users are impacted
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
8
The Mediation Pack provides the capability to extract performance data and metrics from the
following sources
bull SAP Hana DB
bull CA APM (Introscope)
bull MicroFocus NNMi
bull AWS Cloudwatch
bull Pivotal Cloud Foundry
bull Dynatrace Managed
bull AppDynamics
bull Oracle OEM
and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This
document will cover the installation and setup of Mediation Pack for each of the supported source-
data technologies
The figure shows an overall view of the mediation solution in context
Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files
The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively
any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST
interface
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
9
Prerequisite Information
Item Description
Name Mediation Pack Group 2
Predictive Insights supported
versions
v136 and higher Note IBM PI continues to support the CSV
format for ingestion and so earlier versions of PI ie 133 will
also work)
Predictive Insights Information IBM Operations Analytics (demo trial resources)
httpswwwibmcomus-enmarketplaceit-operations-
analytics
IBM Operations Analytics Community (docs videos
mediation packs) httpsdeveloperibmcomitoa
IBM Operations Analytics Predictive Insights
documentation
httpswwwibmcomsupportknowledgecenterSSJQQ3
Solution Name supported
versions
Min version Max Version
CA APM (Introscope) 9x 107
SAP Hana 10 10
AWS Cloudwatch EC2 REST
API 2016-11-
15
Cloudwatch
111
Same as Min
version
MF NNMi 1040x 1040x
Pivotal Cloud Foundry 233 233
Dynatrace Managed Dec 2018 Jun 2019
AppDynamics 41x 459
Oracle OEM 13 13
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
10
Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack
Group 2
Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip
Supported Operating Systems Linux
RHEL 6x 7x 3264bit
CentOS 6x7x 3264bit
Java Java18+ is required There is a version included in the release
package Note Mediators do NOT run on IBM JVM (the one
delivered with PI at time of writing)
Web Browsers A modern HTML5CSS3Javascript browser Versions of
Chrome Firefox or Safari since 2012 should work Previous
browser versions may not work well or at all
Note Some versions of Internet Explorer (ie 9 or earlier)
have exhibited issues Please consider a more modern
browser to ensure full functionality of the product
Ports Port 7457 (default but configurable) is required by
Mediators
Outbound ports per Mediator Type
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
11
SWP Mediators Overview
The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)
specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are
support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files
2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface
Whilst each Mediator requires configuration specific to its source all the data flows through to a
corresponding Predictive Insights distributor The configurations approach for each of the
sources will be described in this document
The Mediators as part of the Mediation pack usually will be installed on the same host as
Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive
Insights If the Mediation pack is installed on a different host then mechanisms must be put in
place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS
FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond
the scope of this document In the case of REST distribution SWP does not have to be co-
located with Predictive Insights
Key steps to setting up Mediators
bull Installation
bull Configure one or more streams which extract the data from the supported sources
bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing
between CSV and REST output
bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This
should result in Predictive Insights CSV files being produced or JSON data
being posted to the IBM PI REST interface
bull Note Setting upconfiguring IBM PI REST interface is beyond the
scope of this document For details on this please refer to the
corresponding IBM PI documentation
bull In the case of CSV integration configure Predictive Insights for ingestion
using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights
lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not
used for PI REST integration
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
12
Installing and Configuring
Zip file contents
The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following
bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral
bull docs containing the User Guides
o for lsquostreamweaverrsquo (this document)
o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt
bull license containing license files
bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for
o Dynatrace
o DCRum
o Aternity
o Splunk
o ExtraHop
o CA UIM
o MF APM
o NetIQ AppManager
o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version
Install procedure
New Installation
The following section provides information for installing and configuring the Mediation Pack on
your IBM Operations Analytics Predictive Insights system
Normal lsquorpmrsquo conventions are followed
1 Verify Hardware amp Software Requirements
2 Verify user account privilege requirements
i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt
with root privileges This is required to create the Linux service
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
13
ii The installer will install as scadmindb2iadm (to match PI)
3 unzip mediation pack file
unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip
4 Install the bundled JVM
rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm
5 Install SWP Mediators Locate the rpm file in the expanded zip directory
rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-
1noarchrpm
After this you should see an indication of successful installation eg
SWP Mediators are installed by default to
optIBMstreamweaver-ibmpi
6 If you used root account exit and return to lsquoscadminrsquo user
Notes
Installing to an alternative location
You can install to a location other than optIBM as follows
sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-
ibmpi-27ibmpiltbuildgt-1noarchrpm
Running as non-`scadmin` user
As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships
If you wish to run as a non-`scadmin` user
Change ownership of all the files from down eg
chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi
Edit the initd startup script
etcinitdstreamweaver-api
Locate and edit the line specifying the DAEMON_USER
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin
Change the default `scadmin` to be your desired user eg
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
14
Co-existence of SWP-based and Accelerator-based mediators
If you are using the earlier Accelerator-based version along with this SWP-based version
note that they use the same default port setting of 7457 In this situation you should change the
default for SWP to something else You can do this by locating and editing the installed config
file in
optIBMstreamweaver-ibmpi confswpconf
and locate uncomment and change the line with port = 7457 to some other port
Settings for the Core Services services
Settings for the UI Service
ui
How the UI Services binds to a socket
connection
Hostname or IP Address to listen on The default
0000 causes
the REST API to listen on all interfaces
host = 0000
The port number to listen on
port = 7457
to some other free port
Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file
httpport = ltrest-port-numbergt must be one less than UI port this defines the
port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-
number server-side clients of REST service use this to locate it NOTE quotes are important here
Data directories
If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
7
Overview
IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a
system through KPI metric data streams This includes data from various sources including
metrics gathered by industry performance and network monitoring solutions A mediation pack
helps users quickly integrate the KPI data streams by providing a default model to integrate a
particular performance manager
IBM Operations Analytics Predictive Insights Mediation Pack Group 2 known in the rest of this
document as the lsquoMediation Packrsquo provides a pre-built model as a starting point to facilitate data
ingestion Predictive Insights creates a performance model that it uses to detect or forecast
behavior outside the modeled range including generating an alarm when an anomalous behavior
occurs Doing so enables organizations to pro-actively avoid system and solution outages before
users are impacted
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
8
The Mediation Pack provides the capability to extract performance data and metrics from the
following sources
bull SAP Hana DB
bull CA APM (Introscope)
bull MicroFocus NNMi
bull AWS Cloudwatch
bull Pivotal Cloud Foundry
bull Dynatrace Managed
bull AppDynamics
bull Oracle OEM
and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This
document will cover the installation and setup of Mediation Pack for each of the supported source-
data technologies
The figure shows an overall view of the mediation solution in context
Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files
The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively
any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST
interface
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
9
Prerequisite Information
Item Description
Name Mediation Pack Group 2
Predictive Insights supported
versions
v136 and higher Note IBM PI continues to support the CSV
format for ingestion and so earlier versions of PI ie 133 will
also work)
Predictive Insights Information IBM Operations Analytics (demo trial resources)
httpswwwibmcomus-enmarketplaceit-operations-
analytics
IBM Operations Analytics Community (docs videos
mediation packs) httpsdeveloperibmcomitoa
IBM Operations Analytics Predictive Insights
documentation
httpswwwibmcomsupportknowledgecenterSSJQQ3
Solution Name supported
versions
Min version Max Version
CA APM (Introscope) 9x 107
SAP Hana 10 10
AWS Cloudwatch EC2 REST
API 2016-11-
15
Cloudwatch
111
Same as Min
version
MF NNMi 1040x 1040x
Pivotal Cloud Foundry 233 233
Dynatrace Managed Dec 2018 Jun 2019
AppDynamics 41x 459
Oracle OEM 13 13
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
10
Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack
Group 2
Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip
Supported Operating Systems Linux
RHEL 6x 7x 3264bit
CentOS 6x7x 3264bit
Java Java18+ is required There is a version included in the release
package Note Mediators do NOT run on IBM JVM (the one
delivered with PI at time of writing)
Web Browsers A modern HTML5CSS3Javascript browser Versions of
Chrome Firefox or Safari since 2012 should work Previous
browser versions may not work well or at all
Note Some versions of Internet Explorer (ie 9 or earlier)
have exhibited issues Please consider a more modern
browser to ensure full functionality of the product
Ports Port 7457 (default but configurable) is required by
Mediators
Outbound ports per Mediator Type
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
11
SWP Mediators Overview
The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)
specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are
support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files
2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface
Whilst each Mediator requires configuration specific to its source all the data flows through to a
corresponding Predictive Insights distributor The configurations approach for each of the
sources will be described in this document
The Mediators as part of the Mediation pack usually will be installed on the same host as
Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive
Insights If the Mediation pack is installed on a different host then mechanisms must be put in
place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS
FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond
the scope of this document In the case of REST distribution SWP does not have to be co-
located with Predictive Insights
Key steps to setting up Mediators
bull Installation
bull Configure one or more streams which extract the data from the supported sources
bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing
between CSV and REST output
bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This
should result in Predictive Insights CSV files being produced or JSON data
being posted to the IBM PI REST interface
bull Note Setting upconfiguring IBM PI REST interface is beyond the
scope of this document For details on this please refer to the
corresponding IBM PI documentation
bull In the case of CSV integration configure Predictive Insights for ingestion
using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights
lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not
used for PI REST integration
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
12
Installing and Configuring
Zip file contents
The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following
bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral
bull docs containing the User Guides
o for lsquostreamweaverrsquo (this document)
o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt
bull license containing license files
bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for
o Dynatrace
o DCRum
o Aternity
o Splunk
o ExtraHop
o CA UIM
o MF APM
o NetIQ AppManager
o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version
Install procedure
New Installation
The following section provides information for installing and configuring the Mediation Pack on
your IBM Operations Analytics Predictive Insights system
Normal lsquorpmrsquo conventions are followed
1 Verify Hardware amp Software Requirements
2 Verify user account privilege requirements
i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt
with root privileges This is required to create the Linux service
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
13
ii The installer will install as scadmindb2iadm (to match PI)
3 unzip mediation pack file
unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip
4 Install the bundled JVM
rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm
5 Install SWP Mediators Locate the rpm file in the expanded zip directory
rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-
1noarchrpm
After this you should see an indication of successful installation eg
SWP Mediators are installed by default to
optIBMstreamweaver-ibmpi
6 If you used root account exit and return to lsquoscadminrsquo user
Notes
Installing to an alternative location
You can install to a location other than optIBM as follows
sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-
ibmpi-27ibmpiltbuildgt-1noarchrpm
Running as non-`scadmin` user
As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships
If you wish to run as a non-`scadmin` user
Change ownership of all the files from down eg
chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi
Edit the initd startup script
etcinitdstreamweaver-api
Locate and edit the line specifying the DAEMON_USER
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin
Change the default `scadmin` to be your desired user eg
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
14
Co-existence of SWP-based and Accelerator-based mediators
If you are using the earlier Accelerator-based version along with this SWP-based version
note that they use the same default port setting of 7457 In this situation you should change the
default for SWP to something else You can do this by locating and editing the installed config
file in
optIBMstreamweaver-ibmpi confswpconf
and locate uncomment and change the line with port = 7457 to some other port
Settings for the Core Services services
Settings for the UI Service
ui
How the UI Services binds to a socket
connection
Hostname or IP Address to listen on The default
0000 causes
the REST API to listen on all interfaces
host = 0000
The port number to listen on
port = 7457
to some other free port
Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file
httpport = ltrest-port-numbergt must be one less than UI port this defines the
port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-
number server-side clients of REST service use this to locate it NOTE quotes are important here
Data directories
If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
8
The Mediation Pack provides the capability to extract performance data and metrics from the
following sources
bull SAP Hana DB
bull CA APM (Introscope)
bull MicroFocus NNMi
bull AWS Cloudwatch
bull Pivotal Cloud Foundry
bull Dynatrace Managed
bull AppDynamics
bull Oracle OEM
and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This
document will cover the installation and setup of Mediation Pack for each of the supported source-
data technologies
The figure shows an overall view of the mediation solution in context
Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files
The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively
any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST
interface
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
9
Prerequisite Information
Item Description
Name Mediation Pack Group 2
Predictive Insights supported
versions
v136 and higher Note IBM PI continues to support the CSV
format for ingestion and so earlier versions of PI ie 133 will
also work)
Predictive Insights Information IBM Operations Analytics (demo trial resources)
httpswwwibmcomus-enmarketplaceit-operations-
analytics
IBM Operations Analytics Community (docs videos
mediation packs) httpsdeveloperibmcomitoa
IBM Operations Analytics Predictive Insights
documentation
httpswwwibmcomsupportknowledgecenterSSJQQ3
Solution Name supported
versions
Min version Max Version
CA APM (Introscope) 9x 107
SAP Hana 10 10
AWS Cloudwatch EC2 REST
API 2016-11-
15
Cloudwatch
111
Same as Min
version
MF NNMi 1040x 1040x
Pivotal Cloud Foundry 233 233
Dynatrace Managed Dec 2018 Jun 2019
AppDynamics 41x 459
Oracle OEM 13 13
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
10
Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack
Group 2
Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip
Supported Operating Systems Linux
RHEL 6x 7x 3264bit
CentOS 6x7x 3264bit
Java Java18+ is required There is a version included in the release
package Note Mediators do NOT run on IBM JVM (the one
delivered with PI at time of writing)
Web Browsers A modern HTML5CSS3Javascript browser Versions of
Chrome Firefox or Safari since 2012 should work Previous
browser versions may not work well or at all
Note Some versions of Internet Explorer (ie 9 or earlier)
have exhibited issues Please consider a more modern
browser to ensure full functionality of the product
Ports Port 7457 (default but configurable) is required by
Mediators
Outbound ports per Mediator Type
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
11
SWP Mediators Overview
The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)
specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are
support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files
2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface
Whilst each Mediator requires configuration specific to its source all the data flows through to a
corresponding Predictive Insights distributor The configurations approach for each of the
sources will be described in this document
The Mediators as part of the Mediation pack usually will be installed on the same host as
Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive
Insights If the Mediation pack is installed on a different host then mechanisms must be put in
place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS
FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond
the scope of this document In the case of REST distribution SWP does not have to be co-
located with Predictive Insights
Key steps to setting up Mediators
bull Installation
bull Configure one or more streams which extract the data from the supported sources
bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing
between CSV and REST output
bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This
should result in Predictive Insights CSV files being produced or JSON data
being posted to the IBM PI REST interface
bull Note Setting upconfiguring IBM PI REST interface is beyond the
scope of this document For details on this please refer to the
corresponding IBM PI documentation
bull In the case of CSV integration configure Predictive Insights for ingestion
using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights
lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not
used for PI REST integration
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
12
Installing and Configuring
Zip file contents
The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following
bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral
bull docs containing the User Guides
o for lsquostreamweaverrsquo (this document)
o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt
bull license containing license files
bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for
o Dynatrace
o DCRum
o Aternity
o Splunk
o ExtraHop
o CA UIM
o MF APM
o NetIQ AppManager
o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version
Install procedure
New Installation
The following section provides information for installing and configuring the Mediation Pack on
your IBM Operations Analytics Predictive Insights system
Normal lsquorpmrsquo conventions are followed
1 Verify Hardware amp Software Requirements
2 Verify user account privilege requirements
i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt
with root privileges This is required to create the Linux service
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
13
ii The installer will install as scadmindb2iadm (to match PI)
3 unzip mediation pack file
unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip
4 Install the bundled JVM
rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm
5 Install SWP Mediators Locate the rpm file in the expanded zip directory
rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-
1noarchrpm
After this you should see an indication of successful installation eg
SWP Mediators are installed by default to
optIBMstreamweaver-ibmpi
6 If you used root account exit and return to lsquoscadminrsquo user
Notes
Installing to an alternative location
You can install to a location other than optIBM as follows
sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-
ibmpi-27ibmpiltbuildgt-1noarchrpm
Running as non-`scadmin` user
As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships
If you wish to run as a non-`scadmin` user
Change ownership of all the files from down eg
chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi
Edit the initd startup script
etcinitdstreamweaver-api
Locate and edit the line specifying the DAEMON_USER
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin
Change the default `scadmin` to be your desired user eg
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
14
Co-existence of SWP-based and Accelerator-based mediators
If you are using the earlier Accelerator-based version along with this SWP-based version
note that they use the same default port setting of 7457 In this situation you should change the
default for SWP to something else You can do this by locating and editing the installed config
file in
optIBMstreamweaver-ibmpi confswpconf
and locate uncomment and change the line with port = 7457 to some other port
Settings for the Core Services services
Settings for the UI Service
ui
How the UI Services binds to a socket
connection
Hostname or IP Address to listen on The default
0000 causes
the REST API to listen on all interfaces
host = 0000
The port number to listen on
port = 7457
to some other free port
Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file
httpport = ltrest-port-numbergt must be one less than UI port this defines the
port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-
number server-side clients of REST service use this to locate it NOTE quotes are important here
Data directories
If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
9
Prerequisite Information
Item Description
Name Mediation Pack Group 2
Predictive Insights supported
versions
v136 and higher Note IBM PI continues to support the CSV
format for ingestion and so earlier versions of PI ie 133 will
also work)
Predictive Insights Information IBM Operations Analytics (demo trial resources)
httpswwwibmcomus-enmarketplaceit-operations-
analytics
IBM Operations Analytics Community (docs videos
mediation packs) httpsdeveloperibmcomitoa
IBM Operations Analytics Predictive Insights
documentation
httpswwwibmcomsupportknowledgecenterSSJQQ3
Solution Name supported
versions
Min version Max Version
CA APM (Introscope) 9x 107
SAP Hana 10 10
AWS Cloudwatch EC2 REST
API 2016-11-
15
Cloudwatch
111
Same as Min
version
MF NNMi 1040x 1040x
Pivotal Cloud Foundry 233 233
Dynatrace Managed Dec 2018 Jun 2019
AppDynamics 41x 459
Oracle OEM 13 13
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
10
Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack
Group 2
Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip
Supported Operating Systems Linux
RHEL 6x 7x 3264bit
CentOS 6x7x 3264bit
Java Java18+ is required There is a version included in the release
package Note Mediators do NOT run on IBM JVM (the one
delivered with PI at time of writing)
Web Browsers A modern HTML5CSS3Javascript browser Versions of
Chrome Firefox or Safari since 2012 should work Previous
browser versions may not work well or at all
Note Some versions of Internet Explorer (ie 9 or earlier)
have exhibited issues Please consider a more modern
browser to ensure full functionality of the product
Ports Port 7457 (default but configurable) is required by
Mediators
Outbound ports per Mediator Type
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
11
SWP Mediators Overview
The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)
specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are
support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files
2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface
Whilst each Mediator requires configuration specific to its source all the data flows through to a
corresponding Predictive Insights distributor The configurations approach for each of the
sources will be described in this document
The Mediators as part of the Mediation pack usually will be installed on the same host as
Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive
Insights If the Mediation pack is installed on a different host then mechanisms must be put in
place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS
FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond
the scope of this document In the case of REST distribution SWP does not have to be co-
located with Predictive Insights
Key steps to setting up Mediators
bull Installation
bull Configure one or more streams which extract the data from the supported sources
bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing
between CSV and REST output
bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This
should result in Predictive Insights CSV files being produced or JSON data
being posted to the IBM PI REST interface
bull Note Setting upconfiguring IBM PI REST interface is beyond the
scope of this document For details on this please refer to the
corresponding IBM PI documentation
bull In the case of CSV integration configure Predictive Insights for ingestion
using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights
lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not
used for PI REST integration
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
12
Installing and Configuring
Zip file contents
The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following
bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral
bull docs containing the User Guides
o for lsquostreamweaverrsquo (this document)
o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt
bull license containing license files
bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for
o Dynatrace
o DCRum
o Aternity
o Splunk
o ExtraHop
o CA UIM
o MF APM
o NetIQ AppManager
o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version
Install procedure
New Installation
The following section provides information for installing and configuring the Mediation Pack on
your IBM Operations Analytics Predictive Insights system
Normal lsquorpmrsquo conventions are followed
1 Verify Hardware amp Software Requirements
2 Verify user account privilege requirements
i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt
with root privileges This is required to create the Linux service
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
13
ii The installer will install as scadmindb2iadm (to match PI)
3 unzip mediation pack file
unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip
4 Install the bundled JVM
rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm
5 Install SWP Mediators Locate the rpm file in the expanded zip directory
rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-
1noarchrpm
After this you should see an indication of successful installation eg
SWP Mediators are installed by default to
optIBMstreamweaver-ibmpi
6 If you used root account exit and return to lsquoscadminrsquo user
Notes
Installing to an alternative location
You can install to a location other than optIBM as follows
sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-
ibmpi-27ibmpiltbuildgt-1noarchrpm
Running as non-`scadmin` user
As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships
If you wish to run as a non-`scadmin` user
Change ownership of all the files from down eg
chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi
Edit the initd startup script
etcinitdstreamweaver-api
Locate and edit the line specifying the DAEMON_USER
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin
Change the default `scadmin` to be your desired user eg
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
14
Co-existence of SWP-based and Accelerator-based mediators
If you are using the earlier Accelerator-based version along with this SWP-based version
note that they use the same default port setting of 7457 In this situation you should change the
default for SWP to something else You can do this by locating and editing the installed config
file in
optIBMstreamweaver-ibmpi confswpconf
and locate uncomment and change the line with port = 7457 to some other port
Settings for the Core Services services
Settings for the UI Service
ui
How the UI Services binds to a socket
connection
Hostname or IP Address to listen on The default
0000 causes
the REST API to listen on all interfaces
host = 0000
The port number to listen on
port = 7457
to some other free port
Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file
httpport = ltrest-port-numbergt must be one less than UI port this defines the
port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-
number server-side clients of REST service use this to locate it NOTE quotes are important here
Data directories
If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
10
Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack
Group 2
Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip
Supported Operating Systems Linux
RHEL 6x 7x 3264bit
CentOS 6x7x 3264bit
Java Java18+ is required There is a version included in the release
package Note Mediators do NOT run on IBM JVM (the one
delivered with PI at time of writing)
Web Browsers A modern HTML5CSS3Javascript browser Versions of
Chrome Firefox or Safari since 2012 should work Previous
browser versions may not work well or at all
Note Some versions of Internet Explorer (ie 9 or earlier)
have exhibited issues Please consider a more modern
browser to ensure full functionality of the product
Ports Port 7457 (default but configurable) is required by
Mediators
Outbound ports per Mediator Type
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
11
SWP Mediators Overview
The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)
specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are
support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files
2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface
Whilst each Mediator requires configuration specific to its source all the data flows through to a
corresponding Predictive Insights distributor The configurations approach for each of the
sources will be described in this document
The Mediators as part of the Mediation pack usually will be installed on the same host as
Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive
Insights If the Mediation pack is installed on a different host then mechanisms must be put in
place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS
FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond
the scope of this document In the case of REST distribution SWP does not have to be co-
located with Predictive Insights
Key steps to setting up Mediators
bull Installation
bull Configure one or more streams which extract the data from the supported sources
bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing
between CSV and REST output
bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This
should result in Predictive Insights CSV files being produced or JSON data
being posted to the IBM PI REST interface
bull Note Setting upconfiguring IBM PI REST interface is beyond the
scope of this document For details on this please refer to the
corresponding IBM PI documentation
bull In the case of CSV integration configure Predictive Insights for ingestion
using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights
lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not
used for PI REST integration
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
12
Installing and Configuring
Zip file contents
The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following
bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral
bull docs containing the User Guides
o for lsquostreamweaverrsquo (this document)
o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt
bull license containing license files
bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for
o Dynatrace
o DCRum
o Aternity
o Splunk
o ExtraHop
o CA UIM
o MF APM
o NetIQ AppManager
o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version
Install procedure
New Installation
The following section provides information for installing and configuring the Mediation Pack on
your IBM Operations Analytics Predictive Insights system
Normal lsquorpmrsquo conventions are followed
1 Verify Hardware amp Software Requirements
2 Verify user account privilege requirements
i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt
with root privileges This is required to create the Linux service
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
13
ii The installer will install as scadmindb2iadm (to match PI)
3 unzip mediation pack file
unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip
4 Install the bundled JVM
rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm
5 Install SWP Mediators Locate the rpm file in the expanded zip directory
rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-
1noarchrpm
After this you should see an indication of successful installation eg
SWP Mediators are installed by default to
optIBMstreamweaver-ibmpi
6 If you used root account exit and return to lsquoscadminrsquo user
Notes
Installing to an alternative location
You can install to a location other than optIBM as follows
sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-
ibmpi-27ibmpiltbuildgt-1noarchrpm
Running as non-`scadmin` user
As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships
If you wish to run as a non-`scadmin` user
Change ownership of all the files from down eg
chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi
Edit the initd startup script
etcinitdstreamweaver-api
Locate and edit the line specifying the DAEMON_USER
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin
Change the default `scadmin` to be your desired user eg
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
14
Co-existence of SWP-based and Accelerator-based mediators
If you are using the earlier Accelerator-based version along with this SWP-based version
note that they use the same default port setting of 7457 In this situation you should change the
default for SWP to something else You can do this by locating and editing the installed config
file in
optIBMstreamweaver-ibmpi confswpconf
and locate uncomment and change the line with port = 7457 to some other port
Settings for the Core Services services
Settings for the UI Service
ui
How the UI Services binds to a socket
connection
Hostname or IP Address to listen on The default
0000 causes
the REST API to listen on all interfaces
host = 0000
The port number to listen on
port = 7457
to some other free port
Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file
httpport = ltrest-port-numbergt must be one less than UI port this defines the
port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-
number server-side clients of REST service use this to locate it NOTE quotes are important here
Data directories
If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
11
SWP Mediators Overview
The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)
specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are
support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files
2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface
Whilst each Mediator requires configuration specific to its source all the data flows through to a
corresponding Predictive Insights distributor The configurations approach for each of the
sources will be described in this document
The Mediators as part of the Mediation pack usually will be installed on the same host as
Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive
Insights If the Mediation pack is installed on a different host then mechanisms must be put in
place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS
FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond
the scope of this document In the case of REST distribution SWP does not have to be co-
located with Predictive Insights
Key steps to setting up Mediators
bull Installation
bull Configure one or more streams which extract the data from the supported sources
bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing
between CSV and REST output
bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This
should result in Predictive Insights CSV files being produced or JSON data
being posted to the IBM PI REST interface
bull Note Setting upconfiguring IBM PI REST interface is beyond the
scope of this document For details on this please refer to the
corresponding IBM PI documentation
bull In the case of CSV integration configure Predictive Insights for ingestion
using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights
lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not
used for PI REST integration
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
12
Installing and Configuring
Zip file contents
The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following
bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral
bull docs containing the User Guides
o for lsquostreamweaverrsquo (this document)
o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt
bull license containing license files
bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for
o Dynatrace
o DCRum
o Aternity
o Splunk
o ExtraHop
o CA UIM
o MF APM
o NetIQ AppManager
o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version
Install procedure
New Installation
The following section provides information for installing and configuring the Mediation Pack on
your IBM Operations Analytics Predictive Insights system
Normal lsquorpmrsquo conventions are followed
1 Verify Hardware amp Software Requirements
2 Verify user account privilege requirements
i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt
with root privileges This is required to create the Linux service
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
13
ii The installer will install as scadmindb2iadm (to match PI)
3 unzip mediation pack file
unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip
4 Install the bundled JVM
rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm
5 Install SWP Mediators Locate the rpm file in the expanded zip directory
rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-
1noarchrpm
After this you should see an indication of successful installation eg
SWP Mediators are installed by default to
optIBMstreamweaver-ibmpi
6 If you used root account exit and return to lsquoscadminrsquo user
Notes
Installing to an alternative location
You can install to a location other than optIBM as follows
sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-
ibmpi-27ibmpiltbuildgt-1noarchrpm
Running as non-`scadmin` user
As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships
If you wish to run as a non-`scadmin` user
Change ownership of all the files from down eg
chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi
Edit the initd startup script
etcinitdstreamweaver-api
Locate and edit the line specifying the DAEMON_USER
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin
Change the default `scadmin` to be your desired user eg
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
14
Co-existence of SWP-based and Accelerator-based mediators
If you are using the earlier Accelerator-based version along with this SWP-based version
note that they use the same default port setting of 7457 In this situation you should change the
default for SWP to something else You can do this by locating and editing the installed config
file in
optIBMstreamweaver-ibmpi confswpconf
and locate uncomment and change the line with port = 7457 to some other port
Settings for the Core Services services
Settings for the UI Service
ui
How the UI Services binds to a socket
connection
Hostname or IP Address to listen on The default
0000 causes
the REST API to listen on all interfaces
host = 0000
The port number to listen on
port = 7457
to some other free port
Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file
httpport = ltrest-port-numbergt must be one less than UI port this defines the
port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-
number server-side clients of REST service use this to locate it NOTE quotes are important here
Data directories
If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
12
Installing and Configuring
Zip file contents
The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following
bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral
bull docs containing the User Guides
o for lsquostreamweaverrsquo (this document)
o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt
bull license containing license files
bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for
o Dynatrace
o DCRum
o Aternity
o Splunk
o ExtraHop
o CA UIM
o MF APM
o NetIQ AppManager
o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version
Install procedure
New Installation
The following section provides information for installing and configuring the Mediation Pack on
your IBM Operations Analytics Predictive Insights system
Normal lsquorpmrsquo conventions are followed
1 Verify Hardware amp Software Requirements
2 Verify user account privilege requirements
i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt
with root privileges This is required to create the Linux service
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
13
ii The installer will install as scadmindb2iadm (to match PI)
3 unzip mediation pack file
unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip
4 Install the bundled JVM
rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm
5 Install SWP Mediators Locate the rpm file in the expanded zip directory
rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-
1noarchrpm
After this you should see an indication of successful installation eg
SWP Mediators are installed by default to
optIBMstreamweaver-ibmpi
6 If you used root account exit and return to lsquoscadminrsquo user
Notes
Installing to an alternative location
You can install to a location other than optIBM as follows
sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-
ibmpi-27ibmpiltbuildgt-1noarchrpm
Running as non-`scadmin` user
As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships
If you wish to run as a non-`scadmin` user
Change ownership of all the files from down eg
chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi
Edit the initd startup script
etcinitdstreamweaver-api
Locate and edit the line specifying the DAEMON_USER
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin
Change the default `scadmin` to be your desired user eg
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
14
Co-existence of SWP-based and Accelerator-based mediators
If you are using the earlier Accelerator-based version along with this SWP-based version
note that they use the same default port setting of 7457 In this situation you should change the
default for SWP to something else You can do this by locating and editing the installed config
file in
optIBMstreamweaver-ibmpi confswpconf
and locate uncomment and change the line with port = 7457 to some other port
Settings for the Core Services services
Settings for the UI Service
ui
How the UI Services binds to a socket
connection
Hostname or IP Address to listen on The default
0000 causes
the REST API to listen on all interfaces
host = 0000
The port number to listen on
port = 7457
to some other free port
Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file
httpport = ltrest-port-numbergt must be one less than UI port this defines the
port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-
number server-side clients of REST service use this to locate it NOTE quotes are important here
Data directories
If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
13
ii The installer will install as scadmindb2iadm (to match PI)
3 unzip mediation pack file
unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip
4 Install the bundled JVM
rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm
5 Install SWP Mediators Locate the rpm file in the expanded zip directory
rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-
1noarchrpm
After this you should see an indication of successful installation eg
SWP Mediators are installed by default to
optIBMstreamweaver-ibmpi
6 If you used root account exit and return to lsquoscadminrsquo user
Notes
Installing to an alternative location
You can install to a location other than optIBM as follows
sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-
ibmpi-27ibmpiltbuildgt-1noarchrpm
Running as non-`scadmin` user
As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships
If you wish to run as a non-`scadmin` user
Change ownership of all the files from down eg
chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi
Edit the initd startup script
etcinitdstreamweaver-api
Locate and edit the line specifying the DAEMON_USER
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin
Change the default `scadmin` to be your desired user eg
[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
14
Co-existence of SWP-based and Accelerator-based mediators
If you are using the earlier Accelerator-based version along with this SWP-based version
note that they use the same default port setting of 7457 In this situation you should change the
default for SWP to something else You can do this by locating and editing the installed config
file in
optIBMstreamweaver-ibmpi confswpconf
and locate uncomment and change the line with port = 7457 to some other port
Settings for the Core Services services
Settings for the UI Service
ui
How the UI Services binds to a socket
connection
Hostname or IP Address to listen on The default
0000 causes
the REST API to listen on all interfaces
host = 0000
The port number to listen on
port = 7457
to some other free port
Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file
httpport = ltrest-port-numbergt must be one less than UI port this defines the
port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-
number server-side clients of REST service use this to locate it NOTE quotes are important here
Data directories
If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
14
Co-existence of SWP-based and Accelerator-based mediators
If you are using the earlier Accelerator-based version along with this SWP-based version
note that they use the same default port setting of 7457 In this situation you should change the
default for SWP to something else You can do this by locating and editing the installed config
file in
optIBMstreamweaver-ibmpi confswpconf
and locate uncomment and change the line with port = 7457 to some other port
Settings for the Core Services services
Settings for the UI Service
ui
How the UI Services binds to a socket
connection
Hostname or IP Address to listen on The default
0000 causes
the REST API to listen on all interfaces
host = 0000
The port number to listen on
port = 7457
to some other free port
Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file
httpport = ltrest-port-numbergt must be one less than UI port this defines the
port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-
number server-side clients of REST service use this to locate it NOTE quotes are important here
Data directories
If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
15
Installation users
SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups
Starting SWP system
SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service
bull As lsquorootrsquo execute
service streamweaver-ibmpi start
In the usual way running lsquostatusrsquo on the service should give an indication that the main
SWP demon is operational
service streamweaver-ibmpi status
Once the SWP demon is running you use the SWP command line utility to monitor the
status of the child processes
Use lsquoswprsquo to observe the status of the system
bull optIBMstreamweaver-ibmpibinswp status
You should see a response similar to the following
In normal operation the five processes should be in Running state
You can also run the status command in a loop (like Linux `watch`) eg
bull optIBMstreamweaver-ibmpibinswp status -w 3
Stopping SWP system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
16
service streamweaver-ibmpi stop
Upgrade installation
service streamweaver-ibmpi stop
rpm -e streamweaver-ibmpi
Install new version as described above
Deployed directory contents
The main directories are created as part of the installation
optIBMstreamweaver-ibmpi
jre
The bundled JRE
bin
lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line
content-packs
lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components
lib
Library files (jars)
logs
Log files useful for monitoring and debugging system behavior
conf
Configuration files to control SWP
services
Cassandra Kafka and swpallinone (core SWP services)
Configuration files to control streamweaver
dataswp
Extracted metric data is stored under here
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
17
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
18
Accessing the Mediation system
Once installed you can interact with the Mediation system via web-browser By default the
Mediation system listens on port 7457 so for example point your web-browser to
httpltservergt7457
Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list
of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section
Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented
Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform
including particularly log levels
System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)
In the subsequent Mediator-specific sections we will give specific examples of
Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST
You are free to use either distributor in each case
The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should
be presented with a list of available Mediator source types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
19
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
20
AppDynamics Mediator
Overview
Extraction Approach
Metric data is extracted using the AppDynamics REST API
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
AppD Port Defaults
REST 443
Mediator creation and configuration
To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your AppDynamics source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
21
Select Destination
Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single
file stream emerges from each Destination and we donrsquot generally want to mix data from
different Collector instances
Give this destination a unique name This name will also be used as a prefix for the
AppDynamics CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
22
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from AppDynamics is based upon selecting Applications Tiers and metric
paths SWP will extract all metrics for the supplied paths
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
23
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)
when data for the next time interval arrives
You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open
for the specified number of periods (as defined by the Maxium Time Window Per File) This is
useful when you want to wait for late data The trade-off is that the files are only made available
to PI after that lag period - thereby increasing end to end latency
For example if you had 5min files and set the max lag to 2 yoursquod have current file open and
two older files leading to up to 15 min latency
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
24
AWS Cloudwatch Mediator
Overview
This mediator extracts Metric data from AWS Cloudwatch system
Extraction Approach
Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference
Pre-requisites
You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users
The Endpoint regions are shown in the following table For the subsequent SWP configuration
you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account
has access to metric data
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
25
T
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
26
Mediator creation and configuration
To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the Source column to add a new origin
Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for AWS Cloudwatch account
Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least
one Region Multiple regions can be entered by separating each with a comma
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the AWS
CloudWatch environment)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
27
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
28
Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics
in those categories SWP will extract all metrics for those selected metric tables For IBM PI
only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are
selected
The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide
convenient resource groupings If you want to scope your metric extraction to particular tagged
instances specify those tags here as comma separated tagname=tagvalue pairs
For more details on AWS tagging see the AWS documentation
The optimum settings for the other configuration values depend on your AWS accountdata
availability etc The defaults should be sufficient to get you operational
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
29
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
30
CA APM (Introscope) Mediator
Overview
CA APM was formerly known as Wily Introscope
Metric extraction from CA APM is based upon configuring the system to extract data for specific
Hosts Metric data will be collected for all Agents and Processes on each selected host
Extraction Approach
Metric data is extracted using a JDBC interface
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
CA APM Port Defaults
JDBC 5001
Note
The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used
Mediator creation and configuration
To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at
httpltmediationservergt7457
Select Source
and then press lsquo+rsquo on the Sources column to add a new sources
Provide a name for the new source instance
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
31
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your CA APM DB
The Metadata Window Width and Latency control time window of metric data which is
extracted to determine the available monitored hosts for selection
Note It is also possible to configure selection to use regexp expressions which are used directly
in the JDBC queries sent to CA APM Depending on whether or not you check the regexp
button you will be presented with a list of Agents(Hosts) to select from or be required to
provide regexps for Agents and Metric selection - see next section
The TimeZone is that of the CA APM server This must be configured to ensure the metric data
is correctly mapped to UTC
Enter the connection information for the CA APM Host
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the CA
APM host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
32
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name
Give this destination a unique name This name will also be used as a prefix for the caapm CSV
files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
33
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from CA APM is based upon selecting the hosts and SWP will extract all
metrics for that host
Note If you had chosen to configure via regexp the list of hosts panel will be empty
You must provide a set of comma separated regexp pairs - one pair per line in the RegExp
panel eg
User
These are regexps for CA APM Agents and Metrics and are used directly in the queries eg
select from metric_data where agent= and metric=User
See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
34
Select the hosts you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Note Sometimes depending on the responsiveness of the CA-APM system it may take a few
seconds for the list of hosts to appear Clicking in the panel often helps with refresh
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
35
Configuring Distributors
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
36
Dynatrace Managed
Overview
This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api
Extraction Approach
Pre-requisites
The hostport and REST api connection token for your Dynatrace Managed system (see your
Admin andor Dynatrace API - Authentication)
Mediator creation and configuration
To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new origin
Press lsquo+rsquo on the SOURCES column to add a new source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
37
Now provide the connectivity information for your Dynatrace Managed system
Enter your Dynatrace Managed credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
Dynatrace Managed environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to produce IBM PI CSV files
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
38
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Select the desired Dynatrace Managed metric sources
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
39
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
40
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
41
MF NNMi Mediator
Overview
This mediator extracts Metric data from MicroFocusrsquo NNMi system
Extraction Approach
Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo
mechanism - these files are retrieved from the NNMi system via FTP or SFTP
Further info can be found here NNMi CSV export
Pre-requisites
You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
42
SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
43
Mediator creation and configuration
To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
Press lsquo+rsquo on the SOURCE column to add a new source
Select lsquoMFNNMirsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your MFNNMi system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
44
Enter your FTP or SFTP connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the
NNMi environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
Connect to Destination
Now configure the system to be able to connect to the IBM PI REST interface
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
45
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection for MFNNMi requires
bull Configuration for file handling following data extraction
o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default
o Following file retrieval if lsquoDelete files when donersquo is
Checked - files will be deleted deleted from MF NNMi
Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo
Note the lsquodonersquo directory does not typically exist and if intended to be used will need to
be manually created on MF NNMi before successful use
bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface
are the only two choices
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
46
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive
the metric data
Note This should be in upper-case per IBM PI requirements
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
47
Configure Stream
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
48
Oracle OEM Mediator
Overview
Extraction Approach
Metric data is extracted using JDBC
For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api
Oracle OEM Port Defaults
JDBC 1521
Mediator creation and configuration
Select Source
Provide a name for the new source instance
And press lsquoCONTINUErsquo (on lower-right of window)
Connect to Source
Now provide the connectivity information for your Oracle OEM source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
49
Note The supplied Host Port and Database Name are used in the construction of the JDBC url
jdbcoraclethin$lthostgtltportgtltdatabaseNamegt
Select Destination
Now select the destination
Give this destination a unique name This name will also be used as a prefix for the Oracle OEM
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
50
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from Oracle OEM is based upon selecting from the available Target Types All
metrics for the selected Target Types within the configured Data Time Window will be
extracted
Select the extraction schedule (Schedule Name)
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
51
Configuring Distributors
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Save Stream
Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)
You should return to Weaver screen and see the stream rendered as below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
52
Pivotal Cloud Foundry Mediator
Overview
This mediator extracts Metric data from Pivotal Cloud Foundry system
Extraction Approach
Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The
nozzle an executable program when run registers for receipt of PCF metrics and as those
metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka
system
For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles
Pre-requisites
For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)
bull UsernamePassword
bull Email User namePassword
bull PCF hostport info
These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need
bull lsquoUAArsquo (see Pivotal Admin) host
bull usernamepassword
These will be supplied to the nozzle program
Kafka
PCF
Collector
Distributors Nozzl
e
Metricss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
53
Overall Setup
Configuration for PCF metric collection involves two main steps
bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the
other mediators above)
bull Configure and start the Nozzle
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
54
Mediator creation and configuration
To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new Source
Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your PivotalCF system
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
55
Enter your PCF credentials and connection details
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the PCF
environment)
Select Destination
Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be
chosen instead
Press lsquoCONTINUErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
56
Connect to Destination
Now configure the system to produce IBM PI CSV files
Select Data Types
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
57
Metric selection from PCF requires setting the following
bull Data extraction schedule
bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following
o PCF has two kinds of metrics which are
lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the
containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics
Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics
o All lsquoPlatformrsquo metrics are collected
o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong
to Spaces and only metrics for Applications within the selected Spaces will be collected
bull Kafka settings
o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is
for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the
use of an external Kafka system if desired
o The Application and Container Metric Topics are the names of the kafka topics which the
Collector looks to retrieve the two types of metrics The default settings match those of of the
supplied Nozzle configuration and should need to be changed
o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka
by the nozzle It will cause the Collector to continually extract data from the beginning of the
specified Kafka topics This can be useful in testing but in general the default (unchecked) should
left as is
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
58
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
59
For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
Press lsquoSAVE STREAMrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
60
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
61
Nozzle Setup
The nozzle executable and associated configuration file can be found in
optIBMstreamweaver-ibmpicontent-packspivotalcfkafka
There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS
For the typical IBM PI deployment the LinuxCentOS one is used namely
kafka-firehose-nozzlelinuxamd64
The associated configuration file is
kafka-nozzletoml
The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI
Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file
(Usually something like wssdopplerlthostgt443)
(Usually something like httpsuaahost)
Save the file
Operation
Once the nozzle configuration file has been prepared and streamweaver service has been
started then the nozzle itself can be started
kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
62
Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo
for metrics Selected metrics will then be delivered to the Kafka instance running within
SWP
Once running you should see output to the nozzle console similar to the following
Note The nozzle is a self-contained executableprocess - you can arrange to run
this from the command line directly as described or as a lsquoservicersquo or otherwise - that
service setup is beyond the scope of this document
To process those PCF metrics and convert them to PI CSV files or deliver them to the
REST if the end-to-end SWP mediator stream must be startedrun in the usual way
To stop processing of PCF metrics
Stop the Mediator stream
Stop the Nozzle (Ctrl-C or otherwise kill the running process)
Note PCF does not persist metrics As metrics are gathered in real-time they are
delivered to via nozzles to any subscribers If the nozzles are not running to collect the
data the data is simply discarded (by PCF) Therefore minimally you should keep the
nozzle and the streamweaver-ibmpi service running to maximize your collection of data
The swpkafka is not intended for long term storage of collected data in the same manner
as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer
the metric data (depending on Kafka topic retention configuration)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
63
SAP Hana Mediator
Overview
This mediator extracts Metric data from SAP Hana database for the user selected tables
Extraction Approach
Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC
SAP Hana Port Defaults
JDBC 30015
Pre-requisites
This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately
JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then
tar xfz hanaclient-23106-linux-x64targz
cd clientclient
tar xfz JDBCTGZ
The driver file is ngdbcjar
Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib
SWPMediators will require a restart to load this driver
Mediator creation and configuration
To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at
httpltmediationservergt7457
Create new source
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
64
From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source
Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo
Provide a name or just go with the default
And press lsquoCONTINUErsquo
Connect to Source
Now provide the connectivity information for your SAP HANA DB system
Enter the connection information for the SAP HANA database
Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can
continue If the connection check failed you will be alerted and you should make whatever
corrections necessary to the configuration (typically credential issues or reachability to the SAP
HANA host)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
65
Select Destination
Now select the destination in this case only IBM PI (CSV) is offered as an option You should
almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from
each Destination and we donrsquot generally want to mix data from different Collector instances
Give this destination a unique name This name will also be used as a prefix for the saphana
CSV files sent to PI
Press lsquoCONTINUErsquo
Connect to Destination
Select Data Types
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
66
Note the items in the right column (Collectors amp Distributors) will not appear until you have
check the lsquoMetricrsquo box
Configuring Collectors
Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract
all metrics for those selected metric tables
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
67
Select the Metrics Schedule This controls how often extraction attempts are made and is
typically set to the same time as the downstream Predictive Insights aggregation interval
Select the tables you want to extract metric data for (just use the mouse pointer to select as many
hosts as you want)
Specify the Data Window and Data Latency as desired
Then press lsquoCREATE COLLECTORS AND CONTINUErsquo
Configure Distributors
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
68
Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)
before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)
when data for the next time interval arrives
Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo
Configure Stream
Then name the stream if you want (or just accept the name provided which is assembled from
Origin and Destination names)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
69
Press lsquoSAVE STREAMrsquo
You should return to WEAVER and see the stream rendered similar to below
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
70
Operations
Starting metric extraction
Select the collector (or distributor) of the stream you want to start T
Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators
Then slide the slider to the right to start metric collectionextraction
Stopping metric extraction
To stop extraction move the slider to the left
Running single extractions
Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
71
From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup
Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job
Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the
stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
72
Editing an existing stream
If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)
You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get
You can navigate through the panels by clicking on the green chevrons (eg the highlighted
green lsquoCOLLECTORrsquo in the above image)
In some cases you may wish to edit the underlying configuration directly in JSON To access the
JSON move the lsquoEdit JSON slider to the right
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
73
When you have completed your editing press lsquoSAVE amp CLOSErsquo
Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON
edit window then lsquoSAVE amp CLOSErsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
74
Predictive Insights CSV Output
All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive
Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what
is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions
etc associated with these files see the main Predictive Insights documentation
File details
Files are output to separate directories corresponding to each up-stream distributor
The general file path pattern is as follows
optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata
Where N is the ID of the distributor which produces the data
To determine this number go to the WEAVER panel click on the SOURCES-side component
AppD in this example and press the gear icon (to the left of the startstop slider)
Conceptually this reveals the configuration screen for the entire end2end channel
Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel
(you may need to scroll down to see it)
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
75
Selecting that will reveal the configuration for this component
Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find
the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
76
In this case the path to data from this distributor is
And examining the directory contents at that path you should see the standard PI csv files (once
you have been runningextracting data)
Timestamps
Timestamps on filenames and in the data itself are in UTC and follow the Java
SimpleDateFormat spec
yyyyMMddHHmmss
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
77
Predictive Insights lsquomodelrsquo files
A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and
these files can be used as a starting point for configuring the Predictive Insights lsquoMediation
Toolrsquo
The model files are located in the directory
optIBMstreamweaver-ibmpicontent-packsibmpicsv
Each one of these files can be loaded into the mediation tool by copying to the Predictive
Insights lsquoMediation Toolrsquo workspace and relevant project eg
cp optIBMstreamweaver-ibmpicontent-
packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo
Then when you refresh the view in the Mediation Tool you should see the new pamodel
alongside any other models you may have eg
You must customize the usual items eg File Path for your particular environment
Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial
mapping for each for the supported technologies These are a starting point You will likely
have to change them for your particular deployment context
Group information
If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the
examples above then that will match the group defined in the corresponding pi model file
However if your Distributor instance has a different name then yoursquoll need to create a new
group using the example one as initial guidance This is the preferred approach
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
78
An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the
underlying XML file For example if we wanted to have a different group for appdynamics
vi homescadminworkspaceltmyprojectgtappdynamicspamodel
Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg
ltnamegtCAAPMltnamegt
ltgroupsgt
ltnamegtcaapmltnamegt
lttimestampKeygt
Also locate and edit here and similarly change lsquoallrsquo to your new group name
ltsourcesgt
ltnamegtcaapm-caapmltnamegt
ltDataEntitySourcegt
Obtaining metric names for PI Mediation tool
The set of actual metrics is highly dependent on what data is actually extracted from the sources
you will also need to add the metric names The quickest way to do this is to first use the
Mediators to extract representative set of CSV files and from those extract the metric names to a
text file Then use the Predictive Insights Mediation tool to import those metrics For example in
an output file from CA APM you would have CSV files with the following header
TimestampmeasurementValuetargetNameNodemeasu
rementNameunitField1ValueField2ValueField3Val
ueField4ValueField5Value
lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a
set of these output CSV files and extract metricNames from this column
bull CA APM - Column 11
bull SAP HANA - Column 4
The following would achieve this
cd pathtoCSVfiles
grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u
You can do similar for each of both supported file types adjusting the column number
appropriately to select the desired lsquometricrsquo column
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
79
Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo
button
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
80
Predictive Insights REST Output
When configured to deliver data to the IBM PI REST distributor a running channel will POST
data to the IBM PI REST service
No further configuration is necessary
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
81
Logs
Logs are written to a number of directories under
optIBMstreamweaver-ibmplogs
The main SWP one is swpallinonelog with logs from the other components (eg kafka) also
appearing the same directory
These will often be the first thing that Support will look for when investigating an issue
Changing log levels
Click on lsquoCONFIGURATIONrsquo on the right-hand panel
The right-hand panel allow you to change log levels for various subsystems - left-click on LOG
LEVELS
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
82
Then you can change the log-levels for individual components
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
83
Support
To get assistance from the IBM support team you may open a Problem Management Record
(PMR)
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM
Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo
Describe the problem encountered providing the following information that must be attached to
the PMR
Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics
Advance Insights
Product Version 136 or higher
Mediation Pack Mediation Pack Group 2
Mediation Pack Version 15
Include the following
bull Sample extracted CSV files which are being ingested into Predictive Insights
bull Data source connectivity status screenshot
bull Log files
o Predictive Insights log files ($PI_HOMElogtopicname) when the
problem occurred while using the mediation pack
o Mediation pack log files ( optIBMstreamweaver-ibmpilog)
bull Run the $PI_HOMEbincollectsh and return collected info
bull A screen shot of the error message that is generated while using the mediation
pack
bull A specific use case scenario(s) where problem occurred while using the
mediation pack
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
84
Appendix 1 SAP HANA metrics
The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
85
HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
86
Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
87
INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
88
reg
copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to
IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA
For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
89
IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed
Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs
Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy
Trademarks
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-
IBM Operations Analytics Predictive Insights Mediation Pack Group 2
90
IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies
- Mediation Pack Group 2
- Contents
- Overview
-
- Prerequisite Information
-
- SWP Mediators Overview
-
- Key steps to setting up Mediators
-
- Installing and Configuring
-
- Zip file contents
- Install procedure
-
- New Installation
- Notes
-
- Installing to an alternative location
- Running as non-`scadmin` user
- Co-existence of SWP-based and Accelerator-based mediators
- Data directories
- Installation users
-
- Starting SWP system
- Stopping SWP system
- Upgrade installation
-
- Deployed directory contents
-
- optIBMstreamweaver-ibmpi
- jre
- bin
- content-packs
- lib
- logs
- conf
- services
- dataswp
-
- Accessing the Mediation system
-
- AppDynamics Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- AWS Cloudwatch Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- CA APM (Introscope) Mediator
-
- Overview
- Note
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Configure Stream
-
- Dynatrace Managed
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new origin
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- MF NNMi Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Oracle OEM Mediator
-
- Overview
- Mediator creation and configuration
- Select Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configuring Distributors
- Save Stream
-
- Pivotal Cloud Foundry Mediator
-
- Overview
- Extraction Approach
-
- Pre-requisites
-
- Overall Setup
- Mediator creation and configuration
- Create new Source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
- Nozzle Setup
- Operation
-
- SAP Hana Mediator
-
- Overview
- Extraction Approach
-
- SAP Hana Port Defaults
- Pre-requisites
-
- Mediator creation and configuration
- Create new source
- Connect to Source
- Select Destination
- Connect to Destination
- Select Data Types
- Configuring Collectors
- Configure Distributors
- Configure Stream
-
- Operations
-
- Starting metric extraction
- Stopping metric extraction
- Running single extractions
- Editing an existing stream
-
- Predictive Insights CSV Output
-
- File details
- Timestamps
-
- Predictive Insights lsquomodelrsquo files
-
- Group information
- Obtaining metric names for PI Mediation tool
-
- Predictive Insights REST Output
-
- Logs
-
- Changing log levels
-
- Support
- Appendix 1 SAP HANA metrics
-