sap copy automation tool - enterprise data cloud …€¦ · · 2017-10-27sap copy automation...

COPY AUTOMATION TOOL FOR SAP

IMPLEMENTATION GUIDE

Version 2.3

May, 2018

Pure Storage Proprietary Information 2 | 33

Copy Automation Tool for SAP Implementation Guide


INTRODUCTION

ABSTRACT

This document describes installation and configuration of the SAP CAT software for SAP systems on:

• UNIX (including Linux) servers, using either Oracle or SAP HANA as the DBMS.

• Windows servers using MSSQL Server as the DBMS.

LICENSE AGREEMENT

Use of SAP CAT is subject to the Pure Storage EULA for Plugin / Adaptor /

Provider / SDK / Management Pack license agreement.

The Plugin EULA can be found on the Product End User Page:

http://www.purestorage.com/legal/productenduserinfo.html

A text version of the EULA is also included in the software archive.

REPORT HISTORY

Version Date Author Description

1.0 01-Sep-2017 Expertum Initial version

1.1 13-Oct-2017 Expertum Updates to:

• prerequisites

• configuration file storage_<SID>

• conductor and client script: ssh configuration changes

1.2 24-Oct-2017 Expertum • Added Appendix: SSH setup

• Added Oracle prerequisite

1.3 25-Oct-2017 Expertum • Added sample config files

• Added License Agreement

1.5 17-Jan-2018 Expertum • Major architecture change: Purity//Run is used as a central machine.

• Initial Windows version

2.0 22-Mar-2018 Expertum • Initial SAP HANA version

• SAPCAT on Docker

2.1 20-Apr-2018 Expertum • Replaced the NFS share with Samba/CIFS

• Rebranding to CAT for SAP

2.2 02-May-2018 Expertum • Design update

http://www.purestorage.com/legal/productenduserinfo.html



• Grammar review

2.3 03-May-2018 Expertum • Updated HANA Setup

• Updated sequence file format

• Added systemcopy parameter



TABLE OF CONTENTS

1 Quick Start .................................................................................................................... 6

1.1 Installation and configuration ............................................................................ 6

1.2 Key terms ........................................................................................................ 6

2 Prerequisites ................................................................................................................. 7

2.1 Software versions ............................................................................................. 7

2.2 Languages ....................................................................................................... 8

2.3 Oracle ............................................................................................................. 8

2.4 FlashArray volumes .......................................................................................... 9

3 Infrastructure ................................................................................................................ 9

3.1 Hosts ............................................................................................................... 9

3.2 SAPCAT container ............................................................................................. 9

3.3 Software directory ........................................................................................... 10

3.4 Shared area .................................................................................................... 10

3.5 Security and access ......................................................................................... 11

4 Setting up SAP CAT..................................................................................................... 12

4.1 Software installation ........................................................................................ 12

4.2 Configuration .................................................................................................. 15

5 Running CAT for SAP .................................................................................................. 29

5.1 Conductor and client script ............................................................................... 29

5.2 Calling systemcopy .......................................................................................... 31

6 Appendix .................................................................................................................... 32

6.1 Password-less SSH setup.................................................................................. 32

6.2 Errors ............................................................................................................. 33

1 QUICK START

1.1 INSTALLATION AND CONFIGURATION

This section contains brief instructions for setting up and running the first system copy.

1. Check that the SAP systems, the database and the operating systems meet the prerequisites specified in section 2.1.

2. For Linux, ensure that the Korn shell (ksh) is installed. For Windows, ensure that PowerShell 3.0 and Microsoft .NET Framework 4.5 are installed. ( 2.1.3)

3. Determine the directories (shared Docker volumes) to be used by the system copy process (3.4)

4. Configure password-free SSH for Unix systems to enable fully automated system copies (

3.5)

5. For each Windows system, run the SAP Host Agent setup.ps1 script. For UNIX systems,

configure the mount in /etc/fstab.( 4.1)

6. Create a sequence file for each copy path using the sequence file delivered with the software as a template. There is usually no need to change the context list, but system IDs, host names, user names, and possibly the path to the SAPCOPY script must be adapted. Section 4.2.2 describes the sequence file.

7. Create a sapcopy.paths file (one file for all copy paths) or use the file delivered with the

software as a template. Section 4.2.6. describes the paths file.

8. Review the sapcopy.properties file and adapt as necessary. Section 4.2.7 describes the

properties file and the properties it contains. Properties that typically must be changed are dirshare and listener_name.

9. Create the clients.conf file (one file for all copy paths) or use the file delivered with the

software as a template. See 4.2.3 for more information.

10. Create storage_<SID> files containing FlashArray configuration information for all source

and target systems involved in copy processes. Section 4.2.8 describes storage configuration files.

1.2 KEY TERMS

The following terms are used throughout this document and by the software:

• A copy path is a system copy identified by the source and target systems. For example, a SAP ECC production system regularly copied to both an ECC test system and an ECC training system and a CRM production system regularly copied to a CRM test system define three copy paths:

o ECC production to ECC test

o ECC production to ECC training

o CRM production to CRM test

• A context is a major phase of a system copy. An end-to-end copy is a series of contexts. Examples of contexts are: pre-copy preparation of target, backup of source system, restore/recovery of target database, post-processing in target system.

• An action is a step in a context. Examples of actions are: exporting tables from the database, stopping SAP, stopping Oracle, etc. Each context is made up of a series of actions.

2 PREREQUISITES

2.1 SOFTWARE VERSIONS

The SAP installation and underlying database and server platform must meet the following prerequisites.

2.1.1 SAP

SAP system type ABAP

SAP NetWeaver version 7.0 and higher (SAP_BASIS >= 700)

2.1.2 DATABASE

Oracle DBMS 11.2.0.3 and higher

Microsoft SQL Server 2012 and higher

HANA 2.0 (Multi-tenant: only one tenant and system db)

The same schema name must be used in the source and target databases.

2.1.3 OPERATING SYSTEM

Linux SAP CAT supports all Linux distributions and versions supported by SAP according to the Product Availability Matrix (PAM)

Solaris

Solaris 10 and higher

AIX

(TBD)

Utilities Unix utilities:

• id

• awk

• sudo

• sed

Windows Windows Server 2008 R2 and higher

Utilities Required:

• PowerShell 3.0 or higher

• Microsoft .NET Framework 4.5 or higher

• SAP Host Agent

2.2 LANGUAGES

Version 1.5 and newer versions of CAT support both Unix/Linux and Windows. CAT for SAP is written in UNIX Korn shell (ksh) and PowerShell scripts. The Windows version also includes a batch script to

allow communication via the SAP Host Agent.

NOTE

The Korn shell is not always installed on Unix systems by default. If it is not present, use the Linux distribution’s installation tool to install the ksh package.

There is no compiled (C) code, Java code or code developed in any other scripting language.

2.3 ORACLE

The SAP CAT script automatically copies the Oracle database data files from the source to the target system. To recover the target Oracle database, the script requires information collected on the source system, including the collected Oracle archive files.

The names of these files are generated based on 2 Oracle parameters, Pure Storage highly recommends setting these as recommended in SAP note 1431798:

LOG_ARCHIVE_DEST_1 'LOCATION=<SAPDATA_HOME>/oraarch/<sid>arch'

LOG_ARCHIVE_FORMAT %t_%s_%r.dbf

This for both source and target systems.

For example, for SID ABC:

alter system set log_archive_dest_1 = 'LOCATION=/oracle/ABC/oraarch/ABCarch

SCOPE=BOTH;

alter system set LOG_ARCHIVE_FORMAT = '%t_%s_%r.dbf' SCOPE=spfile;

If different naming conventions are used, CAT for SAP may not be able to identify the needed files.

https://launchpad.support.sap.com/#/notes/1431798)

2.4 FLASHARRAY VOLUMES

During a system copy SAP CAT overwrites the FlashArray target volume(s) with snapshots of the source volume(s) contents. This implies the following prerequisites for FlashArray volumes:

• Each copied volume should contain only database data files. Because copied volumes are overwritten on the target, all other files, including software installations (database, SAP, etc.) should reside on volumes that are not part of the copy.

• All source volumes containing database data files should be in the same FlashArray protection group. The protection group must be defined in the storage file for the source system (see 4.2.8).

• Each source volume must be mapped to one target volume. Source and target volumes should have a 1-to-1 relationship (see 4.2.8).

Remark, it is possible to copy a source system that has more FlashArray volumes than the target system. To do this, create empty missing target volumes and map them to the corresponding source volumes in the target system’s storage configuration file.

3 INFRASTRUCTURE

3.1 HOSTS

For a given copy path, i.e. a system copy from a specific source system to a specific target system, five systems are involved:

1. The SAPCAT Docker host.

2. The Pure Storage FlashArray

3. The database server of the source system

4. The database server of the target system

5. The primary application server of the target system

If the database and central instance of the target system run on the same server, only three hosts would be involved. The copy scenario delivered with the SAP CAT software assumes that these hosts are separate, but the software can also be used in a single-host target environment.

NOTE

The copy process does not access the primary application server of the source system.

3.2 SAPCAT CONTAINER

CAT for SAP version 2.0 is delivered as a Docker image that can be used to start a container on the Purity//Run machine. It makes it possible to run CAT from a central location with minor setup steps.

3.3 SOFTWARE DIRECTORY

CAT for SAP scripts and configuration files occupy less than 2 MB. A copy operation creates about 2 MB of logs, control files, and temporary files. Pure Storage recommends removing these files periodically.

The CAT scripts are delivered in a Docker container, but need to be ran on each machine.

This facilitates software setup and management by maintaining a single central copy. Ensure that the Docker volumes containing the configuration files are mapped locally on the Docker host to keep them persistent (see 4.1).

3.4 SHARED AREA

A copy from a given source to a given target requires a directory shared between the source and target servers. This is achieved by the SAPCAT Docker container, it offers a shared file system. The CAT processes on all servers involved in the copy process read and write files in this directory.

By default, this shared area is provided by the SAPCAT Docker image as a Samba share.

CAT for SAP uses the shared area mainly for two types of files:

1. Non-persistent files shared by source and target during a copy

2. Backups of source database archive logs needed for database recovery on the target.

Non-persistent files in the shared area are small, typically 1-2 MB per copy. However, Oracle archive log backups may consume a substantial amount of space, which is difficult to predict. CAT for SAP compresses these backups (GZIP)to reduce their size, typically by 70-80%. Pure Storage nevertheless recommends 2-4 GB of free space in the shared area for this purpose.

Figure 11 illustrates the software directory and shared area and the servers requiring access to it. Note that the FlashArray accesses neither; all operations on the array are initiated from the SAPCAT Docker container.

Figure 1: Software directory and shared area

3.5 SECURITY AND ACCESS

The system copy process is controlled by a central "conductor" script, which starts actions on the different servers. The conductor script runs in the SAPCAT Docker container and connects to Unix servers via Secure Shell (SSH) or to Windows systems via the SAP Host Agent with HTTP(S).

A fully automated (-auto or -fullauto option) system copy in a Windows environment requests all

passwords when the conductor script starts. The script encrypts the passwords with a temporary key and stores them in a temporary file. The temporary password file cannot be used for the next refresh.

Fully automatic system copies in UNIX environments require password-free SSH logins from the user running the conductor script to the server-user combinations listed below:

Server User(s)

Source database server User authorized to connect to the database. For Oracle, as SYSDBA (ora<sid> or <sid>adm).

FlashArray Administrator authorized to execute snapshot commands

Target database server User authorized to connect to the database.

For Oracle, as SYSDBA (ora<sid> or <sid>adm).

Target application server <sid>adm

4 SETTING UP SAP CAT

4.1 SOFTWARE INSTALLATION

The application is delivered as a Docker image in a compressed archive (tar extension). After starting the container, the CAT for SAP scripts ares offered via a Samba share. Three important folders are made available as a volume to make them persistent: etc, log and share.

On Unix/Linux machines it is necessary to mount the share SAPCAT on the local mountpoint /mnt/sapcat. The mountpoint used must be the same on each server.

On Windows hosts, run the setup.ps1 script located in the Samba share to install the necessary

scripts onto the SAP Host Agent.

The Docker image provides the Samba services. Therefore, the following ports must be available for mapping on the Docker host:

• 137, 138, 139, and 445 (Samba)

Use the following command to import the image on the Docker host:

docker load --input sapcat2.0.tar

Create a Docker container from the image:

mountpoint="<SAPCAT persistent location>"

docker run -itd --privileged --name=SAPCAT \

--volume=$mountpoint/etc:/mnt/sapcat/etc \

--volume=$mountpoint/log:/mnt/sapcat/log \

--volume=$mountpoint/share:/mnt/sapcat/share \

-p 139:139 -p 445:445 \

-p 137:137/udp -p 138:138/udp purestorage/sapcat

The value of the mountpoint parameter is the location at which to store persistent files on the

Docker host. For example, /home/pureuser/SAPCAT.

4.1.1 STARTING SAPCAT

To start a system copy, log in to the Docker image and start the conductor script at /mnt/sapcat/bin as follows:

docker exec -it SAPCAT bash

Switch to our pureuser user:

su - pureuser

Next you can start the conductor script at /mnt/sapcat/bin:

# cd /mnt/sapcat/bin/

# ./systemcopy ../etc/<sequence file>

4.1.2 UNIX SETUP

Add the following line to the /etc/fstab of each Unix machine participating in the copy to

automatically mount SAPCAT when the system starts:

//<docker hostname>/sapcatunx /mnt/sapcat cifs

username=pureuser,password=<password>,uid=<sidadmuid>,gid=<sapsysgid>,

dir_mode=0777,file_mode=0774,forceuid,forcegid 0 0

The Samba/CIFS share comes with the default credentials:

- User: pureuser

- Password: sapcat

This will mount automatically SAPCAT when the system starts.

After creating the definition in /etc/fstab, login as root and mount the path manually:

mount /mnt/sapcat

4.1.3 WINDOWS SETUP

On Windows systems, install the scripts on the SAP Host Agent to map the Samba share automatically when a CAT instance is started.

With Windows Explorer, navigate to \\<Docker server hostname>\sapcat.

Log in with the default credentials:

• User: pureuser

• Password: sapcat

Remark: Log in to the Docker container and use normal Unix tools to change the pureuser

password.

Start PowerShell and launch the setup script:

cd \\<Docker server hostname>\sapcat

> .\setup.ps1

4.1.4 HANA SETUP

CAT for SAP HANA uses following hdbuserstore keys:

• DEFAULT: connection to the tenant database

• SAPCATSYSTEM: connection to the system database

• CATSYSTENANT: optional connection to the tenant database as an administrator

4.1.4.1 Tenant connection

The DEFAULT key is used on the target database to export/import tables with the native SAP HANA

database tools. To do this, the user linked to the DEFAULT key must have the following permissions:

Database System Privilege

Tenant IMPORT

Tenant EXPORT

Use the following command to list the user that corresponds to the DEFAULT key:

hdbuserstore LIST DEFAULT

4.1.4.2 Optional tenant admin connection

After a system copy, the user of the DEFAULT connection key will have the same permissions as in

the source system. Most likely the source is a production environment and granting the import rights pre-refresh isn’t wanted. Without any permissions you’ll run into problems when importing any native database exports on the target system.

A few optional scripts are defined to allow the complete automation of importing data during the copy process. To use these, you’ll need to configure a hdbuserstore key with a user that has the

permissions to grant EXPORT and IMPORT permissions and enable the scripts in the flow file.

4.1.4.2.1 Connection key

By default, CAT will look for the connection key CATSYSTENANT. This can be changed in the

sapcopy.properties file (see 4.2.7):

default/hdbtenantsys = CATSYSTENANT

Use the following command to create the CATSYSTENANT hdbuserstore key:

hdbuserstore -I SET CATSYSTENANT “hostname:port” “username”

4.1.4.2.2 Scripts

Uncomment the optional scripts in the flow file (see 4.2.5):

[target_prepare]

…

action="hana_grant_import" # Optional: automation of permissions, see

implementation guide

…

[target_sap_postprocess]

…

action="hana_grant_export" # Optional: automation of permissions, see

implementation guide

…

4.1.4.3 System database connection

A hdbuserstore key is required on both source and target database servers to connect to the SAP

HANA system database. On the source system, it is used to change the database backup mode before and after the storage snapshot. The default key SAPCATSYSTEM can be changed for each system

with the CAT property hdbsystemdb in the sapcopy.properties file (see 4.2.7):

default/hdbsystemdb = SAPCATSYSTEM

Use the following command to create the SAPCATSYSTEM hdbuserstore key:

hdbuserstore -I SET SAPCATSYSTEM “hostname:port” “username”

The -i option causes the tool to request a password for the specified username.

Pure Storage strongly recommends creating a user with the following permissions specifically for the SAPCATSYSTEM key:

Database System Privilege

SYSTEMDB BACKUP ADMIN

4.2 CONFIGURATION

This section describes the CAT for SAP configuration files.

All configuration files reside in the Docker volume etc. The etc directory contains examples with the

file extension .sample. These are overwritten each time the Docker container starts.

Copy the examples and edit them to adjust the configuration, e.g.:

cp sapcopy.flow.sample sapcopy.flow

vi sapcopy.flow

4.2.1 OVERVIEW OF CONFIGURATION FILES

Table 1 lists the configuration files used by the current version of CAT for SAP. The first column contains the names of the files in the etc directory. The second column describes each file briefly.

The third column specifies whether adapting the file delivered with the software is mandatory or optional. For example, files that refer to system names (SIDs) must be adapted to the specific configuration. In contrast, other files, such as the list of tables to be truncated in the restored target database, can be used as is in most cases, although they can be edited to meet local needs if necessary.

Some file names contain the source and/or target SID. In the following table, <SSID> and <TSID>

denote the SIDs of the source and target systems respectively.

Filename Description Change Req’d

<SSID>_<TSID>.seq Sequence file Mandatory

clients.conf Client definitions to export/reimport Mandatory

exportsets.conf Tables to export and reimport Optional

sapcopy.flow Flow file (contexts and actions) Optional

sapcopy.paths Authorized copy paths Mandatory

sapcopy.properties System copy properties Mandatory

storage_<SSID> FlashArray configuration data for source system

Mandatory

storage_<TSID> FlashArray configuration data for target system

Mandatory

truncate.conf Tables to be truncated Optional

Table 1: SAP CAT configuration files

4.2.2 SEQUENCE FILE

4.2.2.1 Function

There is a sequence file for each copy path (specific source system to specific target system). Sequence file names are arbitrary (they are referred to explicitly in copy commands), but the recommended location and name is:

<INSTDIR>/etc/<SSID>_<TSID>.seq

For example, with two copy paths, EEPEEQ and CRPCRQ, in installation directory /mnt/sapcat,

the recommended sequence files names are:

/mnt/sapcat/etc/EEP_EEQ.seq

/mnt/sapcat/etc/CRP_CRQ.seq

4.2.2.2 Format Example

The copy path in this example is EEPEEQ. The EEP DB server host name is eepdbhost, the EEQ

DB server host name is eeqdbhost and the EEQ SAP Central Instance server host name is

eeqapphost.

The context names are defined in the flow file (see sapcopy.flow in subsection 4.2.5).

# Sequence file for system copy process using SAP CAT

# 'sourcesid' and 'targetsid' define the system ID of the source and target

sourcesid EEP

targetsid EEQ

# 'sapcopypath' is the full path to the SAPCOPY script

# This must be the same on all servers involved!

sapcopypath /mnt/sapcat/bin/SAPCOPY

# The 'context' entries define the sequence of the copy process for this

source/target combination

# Each entry has the form:

# context connectiontype host user ctx

# connection type: The type of connection used to launch the SAPCAT script

# Currently supported connection protocols:

# - ssh

# - SOAP (Windows servers via the SAPHostAgent)

# host : The host where the conext must run.

# The format depends on the connection type:

# - ssh = hostname

# - soap = http(s)://hostname:port

# - purerun = connection to the Pure Storage FlashArray

# user : user ID for the run

# ctx : name of the context, as defined in the sapcopy.flow file

context ssh eeqdbhost eeqadm target_prepare

context ssh eepdbhost eepadm sourcedb_part1

context purerun eeqdbhost eeqadm dbsnapshot

context ssh eepdbhost eepadm sourcedb_part2

context ssh eeqapphost eeqadm target_stopsap

context ssh eeqdbhost eeqadm target_dbrecover

context purerun eeqdbhost eeqadm target_db_postprocess

context ssh eeqapphost eeqadm target_sap_postprocess

context ssh eeqapphost eeqadm target_final

4.2.3 CONFIGURATION FILE (CLIENTS.CONF)

4.2.3.1 Function

This file controls which client definitions (as configured in SCC4) must be exported from the target before the refresh and re-imported afterwards. This action replaces manual configuration in SCC4 after the refresh.


The following example specifies that the following client definition records should be preserved:

• EEQ: clients 800 and 810

• CRQ: clients 800, 810 and 850

# Configuration file: clients.conf

# This file controls which client definitions (as configured in SCC4)

# must be exported from the target before the refresh and

# re-imported afterwards.

# This action replaces manual configuration in SCC4 after the refresh.

#

# Format of an entry:

# sid:cli

#

# Example:

# QAS:040

# means that SCC4 definition of client 040 in QAS must be

# preserved across refreshes

EEQ:800

EEQ:810

CRQ:800

CRQ:810

CRQ:850

4.2.4 CONFIGURATION FILE (EXPORTSETS.CONF)

4.2.4.1 Function

This file specifies the sets of tables to be exported from the target before the refresh and re-imported into the target afterwards.

During target system preparation, the specified export sets, which contain the tables whose data is to be preserved, are exported. During target post-processing, these sets are re-imported into the refreshed system. Of the specified schemas, only those that actually exist in the target system are re-imported. For example, in ABAP-only systems, export sets that refer to the Java schema are ignored.

4.2.4.2 Format Rules

Each set of tables is a separate section in the file, headed by an exportset entry; parameters for

the section are surrounded by curly brackets ({ }).

Name and arguments Meaning

exportset SETNAME Header for export set SETNAME. The name is arbitrary but must be unique.

method {exp | R3trans} Tool to be used for export and import, either Oracle exp/imp or R3trans. R3trans can only be used for tables in the ABAP schema.

critical {yes | no} If yes, failure of the export terminates the copy process. If no, failure of the export is non-fatal and the copy continues. The default is yes.

schema {abap | java | <name>} Owner of the tables in the list.

The keywords "abap" and "java" represent the SAP ABAP and SAP Java schemas; actual schema names are determined at runtime.

To request export/import of a non-SAP schema, specify its name.

sid *

sid SID1,SID2,…

Specifies the systems on which the export set must be handled. If omitted or set to *, the

export is performed on all target systems. To restrict an export to certain systems only, specify a comma-separated list of system names.

tables TAB1,TAB2,TAB3,…

tables *

Comma-separated list of tables in the export set, or * to export all tables in the schema (because of the number of tables in the ABAP schema, using '*' for the ABAP schema is

inadvisable).


The example below is a excerpt from the export sets file delivered with the software:

exportset SAPLICENSE

{

critical yes

method exp

schema abap

tables SAPLIKEY

}

exportset PROFILES

{

critical yes

method R3trans

schema abap

tables TPFET,TPFHT

}

exportset SCOT

{

critical yes

method R3trans

schema abap

tables

BCSD_RQST,BCSD_STML,SOPR,SXADMINTAB,SXCOS,SXNODES,SXPARAMS,SXROUTE,SXSERV,S

XTELMOOUT

}

exportset STRUST

{

critical yes

method R3trans

schema abap

tables

RSECTAB,SSF_PSE_D,SSF_PSE_H,SSFAPPLIC,SSFAPPLICT,SSFARGS,SSFVKEYDEF,STRUSTS

SL,STRUSTSSLS,STRUSTSSLST,STRUSTSSLT,STRUSTWSSE,STRUSTWSSET

}

WARNING

The license key file (SAPLIKEY) requires export method "exp" (Oracle

export), not "R3trans", because this table is not known in the SAP dictionary.

4.2.4.4 Customization

For example, to preserve customer tables ZDATA1, ZDATA2, and ZDATA3, add the following export

set to the exportsets.conf file:

exportset OURTABLES

{

critical yes

method R3trans

schema abap

tables ZDATA1,ZDATA2,ZDATA3

}

4.2.5 CONFIGURATION FILE (SAPCOPY.FLOW)

4.2.5.1 Function

The flow file defines what CAT for SAP does and where it does it. Specifically, the flow file defines contexts and the actions to be performed in each.

A context is a phase in the system copy process. For example, there is almost always a pre-target context in which data that must be preserved is exported from the target system. Any number of contexts can be defined, provided that together they define an end-to-end system copy.

An action is an activity within a context. Each context therefore consists of a series of activities in a specified sequence. Physically, an action is a shell script.

NOTE

Several flow files are delivered with the CAT software that define a series of contexts that perform an end-to-end copy.

4.2.5.2 Format Rules and Examples

A flow file is divided into sections, each of which corresponds to a context. The first line of each section is the name of the context enclosed in square brackets.

Within a section there are three types of entries:

• A system entry, specifying whether the context operates on the source or target system.

• One or more action entries that specify the names of shell scripts that comprise the

context, optionally followed by command line arguments.

• (Optionally) one or more host entries that specify host(s) on which the context must run.

The example below illustrates a typical flow file section called [copytest]:

[copytest]

system=source

action=always:"first_action"

action="second_action -open"

action="third_action" recovery="fix_error"

action="last_action"

host=EEP:eppdbhost

host=CRP:crpdbhost

The context defined in this section runs on the source and consists of four actions:

• The first_action script is preceded by the qualifier always:, indicating that it is a

mandatory action. Mandatory actions are always executed; they cannot be skipped. Such actions typically collect information needed by the following actions in the context; skipping them would not make sense because the later actions then would be unable to do their work.

• The second_action script receives the command line argument -open.

• The action script third_action has an associated recovery script, described in

subsection 4.2.5.3 below. If third_action fails with a fatal error (i.e., with exit code ≥

64), then CAT invokes the script fix_error to undo the effect of the failed action.

• The last_action.script illustrates the simplest form of action. The action can be skipped

(there is no always: qualifier), it takes no parameters and there is no recovery action.

The section also contains two host entries. These indicate that for source system EEP, the

copytest context must run on the server eppdbhost, and for source system CRP, it must run on

server crpdbhost. If a section contains no host entries, CAT for SAP does not verify that it is

running on the correct host.

4.2.5.3 Recovery actions

If an action script encounters a fatal error, it returns an error code to the SAPCOPY controlling script,

which displays information about the error and stops. In some situations, however, simply stopping the partially complete process is undesirable. For example, if one action shuts down the Oracle database, and a subsequent action fails, it is very likely to be desirable to restart Oracle before SAPCOPY terminates. An action with a recovery script that starts Oracle accomplishes that.

4.2.6 CONFIGURATION FILE (SAPCOPY.PATHS)

4.2.6.1 Function and format

This file specifies authorized copy paths and optional embargo entries. Authorized copy paths protect against accidentally copying to the wrong system. For example, if EEP and EEQ are both SAP ECC systems and CRP and CRQ are SAP CRM systems, then it makes sense to copy EEP to EEQ and CRP to CRQ. However, copying EEP to CRQ or CRP to EEQ would be very unlikely, and EEQ should not be copied to EEP, nor CRQ to CRP.

To ensure that only acceptable paths are used, add these paths to the sapcopy.paths file. The

format for specifying authorized copy paths is source and target SIDs separated by a colon as shown in the examples below:

<source_sid>:<target_sid>

*:<target_sid>

The wildcard in the second example indicates that target_sid may be refreshed from any source.

This may not be appropriate, because, for example, it would allow copying EEP to CRQ.

4.2.6.2 Embargo entries

To prevent normally allowed copies to a specific system, for example:

• A system that has been reserved for training sessions over a 6-week period

• A test system reserved for several months for a critical development project. The test system cannot be refreshed during this period without destroying all development, customization, and data created by the project.

These can also be accomplished by deleting or commenting the copy path, but this is less reliable. The copy path would have to be removed at the start of the critical period and reinstated at its end. A more reliable solution is an embargo entry in the sapcopy.paths file. An embargo entry defines a

period during which copies to a system specified by its SID are not permitted. The form of an embargo entry is:

deny <SID> <fromdate> <todate>

The "from" date and "to" dates are specified as YYYYMMDD.

For example, if copies from production system EEP to test system EEQ are normally allowed, but must be suppressed during the fourth quarter of 2017, for example because EEQ is to be used for a new project, the sapcopy.paths file would contain the following entries:

# Normally allow copies EEP -> EEQ

EEP:EEQ

# Embargo: no EEQ refreshes in 4th quarter of 2017:

deny EEQ 20171001 20171231

4.2.7 CONFIGURATION FILE (SAPCOPY.PROPERTIES)

4.2.7.1 Function and format

This file contains the properties (parameters) to be used for the system copy.

Properties (parameters) take the form of a key-value pair:

property_name property_value

The property name (the key) and its value are separated by one or more spaces.

4.2.7.2 Default and system-specific properties

The name of a property has the form:

prefix/property

Most properties have default values which can be overridden. If the sapcopy.properties file does

not contain a value for a given SAP system, the default value is used. Default properties have the prefix default, whereas the prefix of a system-specific property is the system ID (SID).

In the example below, the default directory for sharing data between source and target system (property name: dirshare) is /usr/sap/trans/tmp, but systems CRP and CRQ use a different

share at /sapshare/data. All other systems use the default share.

default/dirshare /usr/sap/trans/tmp

CRP/dirshare /sapshare/data

CRQ/dirshare /sapshare/data

4.2.7.3 List of properties

The table below does not contain an exhaustive list of properties. As with action scripts these may differ between environments. The properties listed here are the most important ones for CAT in Oracle/UNIX configurations.

The two rightmost columns indicate whether a default (cross-system) value can be set (Dflt = X) and whether system-specific settings that override the default are possible (Sys = X).

Name Meaning Default System

deljoblogs Controls whether old job logs are deleted from /sapmnt/SID/global/*JOBLG directories

during post-processing. Depending on the number of logs, this can be slow.

The parameter takes a numeric value; if the number of job logs is less than or equal to this value, job logs are removed. To suppress removal, set to 0. To force removal regardless of the number of logs, set a high value, such as 999999999.

X X

dirshare Full path to shared directory X X

exportdir Directory for exporting data from target database during pre-processing on target

X X

hdbsystemdb hdbuserstore key used to connect to the SAP

HANA system database. Default: SAPCATSYSTEM

listener_control

See note after table

Stop/start Oracle Listener during target refresh

• false: do not stop/start Listener

• true: stop/start Listener (name must be in listener_name.<SID>)

- X

listener_name

See note after table

Name of Oracle Listener to start/stop during target refresh; only needed if property SID/listener_control is set to true

- X

maxage Maximum age in seconds of the source copy. If it is older than this, the target process does not use it for the refresh.

X X

redosize Redo log size for target database. If not specified, CAT for SAP creates redo logs on the target with the same size as those on the source. This option is useful when the source database has huge redo logs, which are not especially useful and can increase recovery time on the target.

- X

Name Meaning Default System

signalwait Number of seconds that the target process waits for the source process to complete.

X X

sourceops Defines what to do in the target database with the OPS$ users that have come over from the source. Possible values are "drop", "lock" and "keep.

X X

Table 2: SAP CAT properties

NOTE

The listener_control and listener_name properties must be adapted

because they are inherently SID-specific. Other properties may be left at their default settings or changed to system-specific settings. In addition, the default settings may be changed.

4.2.8 CONFIGURATION FILES (STORAGE_<SID>)

4.2.8.1 Function

These files contain settings for the FlashArray snapshot process. Each copy process normally involves two storage configurations:

• File storage_<SSID> for the source system

• File storage_<TSID> for the target system

4.2.8.2 Format

Each record in the storage file has the form:

type:value

Records of a given type may appear more than once in the same file.

The record types are:

• flashArray

Host name or IP address of the Flash Array. This entry must exist on both source and target.

• flashArrayUser

UserID for logging in to the Flash Array; This entry must exist on both source and target.

• pgroup

Name of the protection group. The source system protection group must contain only FlashArray volumes that hold database data files. Match the snapped volumes with the volumes mapped in the storage file of the target system. If more (or fewer) volumes are snapped, the copy will not start. On target systems, a protection group of the whole system can be used to create a snapshot backup before starting the copy procedure.

• fs: File system mount path; exists only on the target side.

NOTE For AIX source systems, this entry should also be created. These are used to:

• Gather the AIX LVM storage information.

• Temporarily freeze the file systems for maximum 60 seconds when making the snapshot.

WARNING

On AIX systems, file system paths are used to derive the physical volumes from the containing volume group (via the logical volume). These are unmounted; varied off and exported on the target system!

Thus, physical volumes may only contain logical volumes related to the database SAP data files. Database data files should therefore be created in a separate volume group on separate dedicated physical volumes/disks.

• volmap_<SSID>_<TSID>

Volume mappings from source to target. Present only on the target.

Volume mappings are the “brain” of CAT for SAP. They specify which FlashArray volumes are to be copied from the source to the target. As a prerequisite, the number of source volumes on should be equal to the number of target volumes.

NOTE For AIX filesystems only:

SAP CAT handles volume management on AIX target systems, including attaching disks and importing the volume groups on the target system based on storage information collected from the source system.

If the source system has more FlashArray volumes than the target system, empty FlashArray volumes must be created on the target system.

The final step is then listing the volume mapping between the volumes of system A and system B in the storage file of the target system.

End of: AIX filesystems

4.2.8.3 Example

Example of a storage file for a source system:

flashArray:10.148.25.2

flashArrayUser:pureuser

pgroup:OracleCRPGroup

Example of a storage file for a target system:

flashArray: 10.148.25.2

flashArrayUser:pureuser

fs:/oracle/SID/sapdata1








volmap_CRP_CRQ:OracleDataVol1 OracleCPDataVolume1





volmap_CRP_CRQ:OracleOrigLogA OracleCPOrigLogA

volmap_CRP_CRQ:SAPOrigLogB OracleCPOrigLogB

volmap_CRP_CRQ:OracleMirrLogA OracleCPMirrLogA

volmap_CRP_CRQ:OracleMirrLogB OracleCPMirrLogB

4.2.9 CONFIGURATION FILE (TRUNCATE.CONF)

4.2.9.1 Function

This file defines the sets of target system tables to be truncated (emptied) after the refresh.

4.2.9.2 Format

An entry in the truncation list has one of the following forms:

abap:TABLE

<schema>:TABLE

The prefix abap refers to tables in the ABAP schema; actual schema names are substituted at

runtime. Tables in non-SAP schemas can be truncated by preceding their names with those of their schemas.

4.2.9.3 Example

The example below lists a set of ABAP tables to be truncated plus two tables in external (non-SAP) schema PERF.

# List of tables to truncate after refresh and before SAP start

# Format:

# abap:TABLE

# java:TABLE

# <name>:TABLE

#

# 'abap' and 'java' are placeholders for the names of the

# ABAP/Java schema owners (actual schmena names are substituted at

# runtime). <name> is the name of a non-SAP schema.

#

# Note: ABAP table DDLOG is *always* truncated, whether

# listed here or not.

abap:DDLOG

abap:SDBAD

abap:SDBAH

abap:SDBAP

abap:SDBAR

abap:DBMSGORA

abap:VBHDR

abap:VBMOD

abap:VBDATA

abap:VBERROR

abap:TPFHT

abap:TPFET

abap:SNAP

abap:TLOCK

abap:RZLLITAB

abap:DBSTATTORA

abap:DBSTATIORA

abap:DBSTATHORA

abap:DBSTAIHORA

abap:TPFID

abap:TPFBA

abap:BTCOMSET

abap:BTCOMSDL

abap:MONI

abap:SWNCMONI

abap:SWNCMONIINDEX

abap:APQI

abap:APQD

PERF:COLLECTOR_INFO

PERF:COLLECTOR_HIST

5 RUNNING CAT FOR SAP

5.1 CONDUCTOR AND CLIENT SCRIPT

To carry out a complete system copy, run a "conductor" script. This script takes as input a sequence file (see 4.2.2) to determine which contexts (phases) must be executed, in what order and on which server. The conductor is a shell script called systemcopy.

Depending on the connection type defined in the sequence file, the conductor connects via 2 possible protocols with the server responsible for executing the actions in that context:

• Secure Shell (SSH) for UNIX

• SOAP (HTTP/S) for SAP Host Agents on Windows

To run a context, the conductor logs in to the appropriate server via the specified protocol and invokes the client script responsible for executing the actions in that context.

5.1.1 UNIX

In general, when the systemcopy script for a CAT run on UNIX systems starts, the following steps

are taken:

• The systemcopy script is started on the SAPCAT Docker container with a given sequence

file.

• For each sequence in the file a CAT instance is started. Depending on the connection type, an instance is started on:

o The SAPCAT Docker container

o A source database or target database/SAP system

The complete connection flow with UNIX systems is shown in Figure 2:

Figure 2: Conductor script systemcopy

The client script, which is called SAPCOPY, receives the name of the context as one of its parameters.

SAPCOPY then looks up that context in the flow file (see 4.2.5), where it finds the list of actions that belong to the context. It then executes one action after the other. This is shown schematically in Figure 3: to execute a context do_xyz, the conductor launches SAPCOPY on the remote host.

SAPCOPY reads the action list of do_xyz from the flow file and runs the actions (scripts) action1,

action2, action3, etc.

Figure 3: Client script SAPCOPY

5.2 CALLING SYSTEMCOPY

5.2.1 WHICH USER?

Any server user that meets the following conditions can run systemcopy:

• Read and write access to the shared directory

• Password-less SSH to all servers involved in the system copy (source DB server, target DB server, target CI server and Flash Array)

• Member of the sudo group

NOTE

Several actions require the sidadm or orasid user to start the sudo

command. For example:

• AIX volume handling

• Run as another user: sudo su –orasid –c “command…”

• mount/umount

5.2.2 SYNTAX

The syntax of the systemcopy command is as follows:

systemcopy [ -dialog | -auto | -fullauto ]

[-flow flowfile]

[ -restart | -norestart ]

seqfile

Command arguments:

-dialog

Run in dialog (interactive) mode. In this mode, systemcopy prompts at the beginning of

every context and SAPCOPY at the beginning of every action.

-auto

Run in automatic mode. In this mode, systemcopy prompts at the beginning of every

context; the context itself, however, runs without interaction.

-fullauto

Run in fully automatic mode. There is no user interaction; the system copy runs from beginning to end without human intervention.

-flow flowfile

Run with the given flow file. Remark: it is mandatory to use the relative path to the flow file (see 4.2.5 for a description of the flow file).

-restart

If the previous system copy on this path failed, this option instructs systemcopy to continue at the point of failure. If the previous copy ended successfully, this option has no effect.

-norestart

If the previous system copy on this path failed, this option instructs systemcopy to start a fresh copy from the beginning. If the previous copy ended successfully, this option has no effect. Note: if neither option is specified and the previous copy failed, then the behavior is as follows:

• Dialog mode: systemcopy prompts interactively for a decision (continue failed copy or

start anew)

• Auto / full auto mode: the script reports an error and stops. Either -restart

or -norestart must be specified.

seqfile

Name of the sequence file for this copy path. This is the only mandatory argument. See section 4.2.2 for a description of the sequence file.

6 APPENDIX

6.1 PASSWORD-LESS SSH SETUP

To avoid password prompts during the execution of SAP CAT, set up a SSH key. The public and private key pair is used to automatically log on on the different systems.

This example sets up user ecqadm to automatically login onto the FlashArray’s pureuser account.

6.1.1 GENERATE PRIVATE AND PUBLIC KEY

If no key pair exists, generate a key pair for the user to automatically login onto the remote system as follows:

ecqadm> ssh-keygen -t ecdsa -b 521

Accept the default path:

Generating public/private ecdsa key pair.

Enter file in which to save the key (/home/ecqadm/.ssh/id_ecdsa):

Just press ENTER when asked for passwords:

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

This generates following output:

Your identification has been saved in /home/ecqadm/.ssh/id_ecdsa.

Your public key has been saved in /home/ecqadm/.ssh/id_ecdsa.pub.

…

Copy the contents of the public key file to a buffer:

ecqadm> cat /home/ecqadm/.ssh/id_ecdsa.pub

Select the output and copy it for example with CTRL+C.

6.1.2 ADD THE PUBLIC KEY TO THE PURE ARRAY

In this step, paste the public SSH key into the FlashArray GUI.

• Log via the web browser into the storage array with the user pureuser.

• Got to: System > User > Me

• Click on Setting (the gear icon)

• Select Update Public Key

• Paste the copied public key. Make sure additional line breaks are removed and click Save.

6.1.3 TEST

Start a ssh session as ecqadm to the array with user pureuser without a password prompt.

For example:

ecqadm> ssh pureuser@<array_name> purevol list

6.2 ERRORS

6.2.1 XSOAP

The SOAP protocol is used to connect with the SAP Host Agent on Windows servers.

This section lists the possible exit/error codes and the possible solutions. The error codes returned are those from the tool curl.

6.2.1.1 xsoap: 6

In this case there is clearly a connection problem:

• Is the hostname known inside the SAPCAT container? Can it resolve the ip address?

• Is there even a connection possible? Firewall? Network range/setup?

6.2.1.2 xsoap: 22

When error 22 is shown, it indicates a possible authorization issue:

• Wrong username and/or password?

Another possibility is that the program failed. Please check the SAP Host Agent and CAT log files.

sap copy automation tool - enterprise data cloud …€¦ · · 2017-10-27sap copy automation...

Documents