configuring a tibco enterprise message service environment ......• setting up linux vm instance on...

TIBCO Software Inc.

Global Headquarters

3307 Hillview Avenue

Palo Alto, CA 94304

Tel: +1 650-846-1000

Toll Free: 1 800-420-8450

Fax: +1 650-846-1005

www.tibco.com

TIBCO fuels digital business by enabling better decisions and faster, smarter actions through the TIBCO Connected Intelligence Cloud. From APIs and systems to devices and people, we interconnect everything, capture data in real time wherever it is, and augment the intelligence of your business through analytical insights. Thousands of customers around the globe rely on us to build compelling experiences, energize operations, and propel innovation. Learn how TIBCO makes digital smarter at www.tibco.com.

Configuring a TIBCO Enterprise Message ServiceTM Fault Tolerant Environment on the Google Cloud Platform This document provides the steps for configuring and testing TIBCO Enterprise Message Service Fault Tolerance with a Linux operating environment on GCP utilizing Zadara Storage

Version 1.0 October 2019 Initial Document

©2019 TIBCO Software Inc. All Rights Reserved. 2

Copyright Notice COPYRIGHT© 2019 TIBCO Software Inc. All rights reserved.

Trademarks TIBCO, the TIBCO logo, and TIBCO Enterprise Message Service are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries. All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only.

Content Warranty The information in this document is subject to change without notice. THIS DOCUMENT IS PROVIDED "AS IS" AND TIBCO MAKES NO WARRANTY, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING BUT NOT LIMITED TO ALL WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. TIBCO Software Inc. shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance or use of this material.

For more information, please contact:

TIBCO Software Inc. 3303 Hillview Avenue Palo Alto, CA 94304 USA


Table of Contents

Configuring a TIBCO Enterprise Message ServiceTM Fault Tolerant Environment on the Google Cloud Platform ......................................................................................................................................... 1

1 Overview ................................................................................................................................ 5 1.1 Zadara Virtual Private Storage Array Support .................................................................................. 5 1.2 Linux Kernel Support ...................................................................................................................... 5 1.3 EMS Version Support ...................................................................................................................... 5 1.4 Prerequisites .................................................................................................................................. 6

2 Compute Engine Instance Creation in GCP .............................................................................. 7

3 Zadara Virtual Private Storage Array Configuration ................................................................. 9

4 Configuring the GCP VM Instances ....................................................................................... 13 4.1 Software updates ......................................................................................................................... 13 4.2 Configuring NFS ............................................................................................................................ 13

4.2.1 IPsec installation and Configuration .......................................................................................... 13 4.2.2 Create the Zadara NFS Mount Configuration ............................................................................ 14 4.2.3 NFS Mount via Fstab ................................................................................................................. 15 4.2.4 Linux Kernel Changes ................................................................................................................ 15

5 Enterprise Message Service Installation and Configuration ................................................... 17 5.1 EMS Installation ............................................................................................................................ 17 5.2 EMS Configuration ........................................................................................................................ 17

5.2.1 Stores.conf ............................................................................................................................... 17 5.2.2 Factories.conf ........................................................................................................................... 18 5.2.3 Tibemsd.conf ............................................................................................................................ 18 5.2.4 Starting the EMS Instances ....................................................................................................... 20

6 Testing EMS Fault Tolerance on GCP with Zadara Storage ..................................................... 21 6.1 EMS Client Application Setup ........................................................................................................ 21 6.2 Performing the EMS Fault Tolerant Test Cases .............................................................................. 22

6.2.1 EMS Process Failure Test .......................................................................................................... 22 6.2.2 Network Failure Test ................................................................................................................ 25 6.2.3 System Failure Test ................................................................................................................... 29


Table of Figures FIGURE 1 - CREATE GCP VM INSTANCE ............................................................................................................................... 7 FIGURE 2 - VPSA GUI ..................................................................................................................................................... 9 FIGURE 3 - CONFIGURING A SERVER IN THE VPSA ................................................................................................................. 10 FIGURE 4 - CREATE SHARED VOLUME EXAMPLE .................................................................................................................... 11 FIGURE 5 – EXAMPLE OF A CREATED VOLUME ...................................................................................................................... 12 FIGURE 6 - ATTACHED SERVERS TO THE VPSA...................................................................................................................... 12 FIGURE 7 - MNT-NFS.MOUNT SERVICE EXAMPLE ................................................................................................................... 15 FIGURE 8 - FSTAB EXAMPLE ............................................................................................................................................. 15 FIGURE 9 - SYSCTL CHANGE ............................................................................................................................................. 16 FIGURE 10 - STORES.CONF EXAMPLE ................................................................................................................................. 18 FIGURE 11 - FACTORIES CONFIGURATION EXAMPLE ............................................................................................................... 18 FIGURE 12 - STARTING AN EMS INSTANCE EXAMPLE ............................................................................................................. 20 FIGURE 13 - CREATE THE SYNC QUEUE ............................................................................................................................... 22 FIGURE 14 - RUNNING THE TIBJMSMSGPRODUCERPERF APPLICATION ....................................................................................... 23 FIGURE 15 - THE STANDBY EMS INSTANCE BECOMING ACTIVE ON SERVER2 ................................................................................ 24 FIGURE 16 - PURGE THE SYNC QUEUE FOR TIBEMSADMIN ....................................................................................................... 25 FIGURE 17 - DROP_NFS.SH SCRIPT .................................................................................................................................... 26 FIGURE 18 - RUNNING DROP_NFS.SH SCRIPT ....................................................................................................................... 27 FIGURE 19 - DISK WRITE ERROR ON SERVER1 ...................................................................................................................... 27 FIGURE 20 - SERVER2 BECOMING ACTIVE AFTER NETWORK FAILURE .......................................................................................... 28 FIGURE 21 - GCP COMPUTE ENGINE/VM INSTANCES PAGE .................................................................................................... 29 FIGURE 22 – STANDBY EMS INSTANCE RECOVERING FROM SYSTEM FAILURE OF PRIMARY ............................................................... 30


1 Overview

The purpose of the document is to provide a guide to install, configure, and run TIBCO Enterprise Message ServiceTM (EMS) in a fault-tolerant configuration on the Google Cloud Platform (GCP) utilizing a Zadara Storage System solution for shared storage. In addition, the document will provide the steps and expected results for testing EMS F/T on GCP with Zadara.

The document will outline:

• Setting up Linux VM instance on GCP • General outline for setting up the Zadara VPSA for the shared file system • Setting up the NFS4 mount on the Linux GCP Compute Engine (VM instance) • Installing and configuring EMS for F/T • Tuning EMS for the GCP environment • Running tests for:

o EMS process failure o Network failure between the GCP VM instance running EMS and the shared storage o Accidental reboot of the GCP VM instance from the GCP Dashboard

1.1 Zadara Virtual Private Storage Array Support

As of the writing of this document, the Google Cloud Platform does not provide a shared storage platform capable of supporting the shared storage requirements required for TIBCO Enterprise Message Service. Zadara provides an Enterprise Storage as a Service Solution (STaaS). Zadara provides a fully managed on premise or in the cloud storage solution that meets the shared storage requirements for EMS when utilizing NFSv4. See https://www.zadara.com and http://guides.zadarastorage.com/vpsa-guide/1711/ for details on accessing a Zadara Virtual Private Storage Array (VPSA). Note: This document does not provide guidance on creating or accessing the VPSA, but will provide guidance on configuring a NAS/NFSv4 for EMS on the VPSA.

1.2 Linux Kernel Support

This document covers the installation and configuration of EMS on CentOS Linux 7.6, which is based on Linux kernel 3.10.0-957. All Linux distributions that are materially equivalent to the above listed Linux version are supported. See https://docs.tibco.com/pub/ems/8.5.0/TIB_ems_8.5.0_readme.txt?id=1 for details.

1.3 EMS Version Support

EMS version 8.5 or later must be used for EMS server installations running in a GCP environment. Any available EMS hot fixes should also be applied.


The Enterprise Edition or the Community Edition of EMS can be used. Download the EE from edelivery.tibco.com , and the CE at https://www.tibco.com/products/tibco-enterprise-message-service.

1.4 Prerequisites

The following prerequisites must be meet before configuring EMS on GCP:

• A Google Cloud Account is required. A Google Cloud Account can be created/accessed at https://console.cloud.google.com.

• The TIBCO Enterprise Message Service Software version 8.5.0 has been downloaded and is available to be uploaded to GCP.

• The Zadara VPSA onboarding has been completed, and the VPSA is available for configuration. See section 1.1 for details.

• The reader of this document is familiar with the following concepts: o The use of Google Cloud Console for configuring Compute Engines o TIBCO EMS installation and configuration o Linux configuration o NFSv4 o Zadara VPSA o Configuration of IPsec for encryption (optional)


2 Compute Engine Instance Creation in GCP

Three GCP Linux VM instances are required. The following steps will outline creating the VM instances in the GCP console. CentOS Linux 7.6 was used for the VM instances. Other operating systems or versions can be used, but are not covered in this document. For the following examples, the GCP region US-EAST-4 is used. Note: TIBCO Enterprise Message Service 8.5 is currently not supported on CentOS/RHEL 8 distributions.

• Log into the Google Cloud Platform (GCP). • On the left hand side, click on Compute Engine, and then on Create Instance.

Figure 1 - Create GCP VM Instance

• Provide the Name. Note the name, as it will be used when configuring the Zadara VPSA • Select the Region • Select the Zone. Each VM instance should be created in a different zone • Select General Purpose, and the desired CPU generation • Select the machine type. For development or non-performance testing, this can be relatively

small, such as an n1-standard-2 (2vCPU/7.5GB memory). For production environments, this should be a large instance based on the number of EMS instances and EMS utilization. GPU is not required

• The recommended boot disk is 20GB if only EMS is being installed • The recommended image is CentOS7. Other images may work, but only CentOS7 has been

tested with this configuration and setup • The Service Account should be Compute Engine default service account • The Access Scope and Firewall depend on individual requirements, but are necessary for

basic EMS access • Click on Create


• Create two additional VMs. Two of these will be used for the EMS servers, while the third will be use for EMS client testing. Note: The EMS client VM instance can be smaller, with less vCPUs and Memory.

• Once completed, note the Name, Internal, and External IP Address or the new compute engines


3 Zadara Virtual Private Storage Array Configuration

The Zadara Virtual Private Storage Array (VPSA) must be configured before completing the setup of the GCP instances for EMS. See section 1.1 on setting up access to the Zadara Virtual Private Storage Array (VPSA). Zadara will provide the IP Address of the VPSA, as part of the onboarding procedure. This will also be the IP address used on the GCP VM instances for the NFS mount. Once there is access to the Zadara VPSA, follow this section to create the basic configuration for the Zadara VPSA with NFSv4 for use as the shared storage for EMS.

• Log into the VPSA using the username and password configured during the initial VPSA setup with Zadara. In the following examples, EMSTEST is the name of the VPSA. Click on Drives under resources, and verify there is at least one drive to configure.

Figure 2 - VPSA GUI

• Next, click on Servers under Resources. Click on Add and then Manual. This will need to be done for the two GCP VMs used as the EMS servers in GCP. Provide:

o The name of the GCP VM instance used for EMS server o The OS type (Linux) o The internal IP Address of the GCP VM instance o Click on Enable IPsec if end to end encryption is required (optional, but

recommended). o Click on Create

• The newly created server should look similar to the following example.


Figure 3 - Configuring a Server in the VPSA

• Create the volume in the VPSA for the NFSv4 mount • Click on Volumes under Resources, and then on Create, and then click on Create NAS

Share. Provide the: o The name for the new volume o The capacity for the volume (in GiB). Must be below the capacity of the drive o Export name for the NFS server. Note the export name, a this will also be used as

part of the NFS mount on the GCP VM instances. o Click on encrypted, if data at rest encryption is required. If IPsec was enabled

previously, it recommended to enable encryption at rest also o Ensure 512 KiB read ahead is selected o The rest of the values can be at their defaults if desired o Click on submit


Figure 4 - Create Shared Volume Example

• Once the new volume is created, click on the details for the newly created volume • Ensure that the all settings defined previously have been enabled


Figure 5 – Example of a created Volume

• Click on Servers from the top section of the VPSA GUI, while still on volumes • Next, click on Attach to Server(s), and click on the two servers previously created, and click

submit. This will allow the two GCP VM instances to be attached to the VPSA via the NFS mount.

• Click on Servers in the details for the volume. The two VM instances should be listed.

Figure 6 - Attached Servers to the VPSA

• The VPSA is now ready for use as the shared storage device.


4 Configuring the GCP VM Instances

The individual GCP VM instances can now be configured. The following sections will outline additional software and changes needed for EMS installation and configuration.

4.1 Software updates

Additional software is required to install and configure EMS and NFSv4 on the CentOS7 VM created in GCP. Not all software needs to be installed all machines, but it simplifies the installation.

• Log into each of the newly created GCP VMs. Follow the GCP documentation for using ssh to connect to each VM. The easiest approach is to click on SSH from the Google Cloud Console as shown in Figure 1.

• There are several packages to install, but all can be installed at once. Use o sudo yum install -y nfs-utils java-devel unzip to install all packages o sudo yum update -y to update all packages

• Repeat on the other VMs

4.2 Configuring NFS

On the two GCP VM instances being used as the EMS servers, NFSv4 must be configured and the shared storage mounted. If IPsec is being used, it must be installed and configured first. If IPsec is not being used, sections 4.2.1 and 4.2.2 can be skipped.

4.2.1 IPsec installation and Configuration IPsec will provide “end to end” encryption of all EMS messages between the EMS server and the Zadara VPSA. Follow this section to install and configure IPsec.

• An additional software package must be downloaded and installed. Openswan is utilized to connect to Zadara VPSAs over IPsec. Use sudo yum install –y openswan to install openswan

• Add the following to /etc/ipsec.conf:

conn zadara-VPSA-NFS type=transport auto=start authby=secret ike=aes128-sha1-modp1024 phase2=esp phase2alg=aes128-sha1 ikev2=never dpddelay=5 dpdtimeout=5 dpdaction=restart


pfs=no salifetime=1h rekey=yes rekeymargin=9m keyingtries=%forever # Always try to connect to VPSA ikelifetime=3h left=<Internal IP Address of the VM instance> leftprotoport=tcp right=<IP Address of the VPSA> rightprotoport=tcp/2049

• The file /etc/ipsec.secrets needs to contain the shared secret. This is the IPSEC key from the VPSA:

o From the Main Dashboard panel of the VPSA on the left hand side, click on System, then Settings.

o In the main screen, click on Security. The IPSEC key is at the bottom of the screen. This is the PSK.

o Edit /etc/ipsec.sec, and modify as shown below:

################### <IP Address of the VM instance> <IP Address of the VPSA>: PSK "<PSK>" ###################

For example: 10.150.0.3 172.24.140.200: PSK "D12340348GF34E26A481D74A1F456789"

• Start IPsec: sudo systemctl restart ipsec • To start after reboots: sudo systemctl enable ipsec • To verify IPsec, use: sudo ipsec whack –status - The output should contain information

that the connection to the Zadara VPSA has been established. Note: If IPsec is being used, Section 4.2.1 must be completed on both GCP instances where the EMS servers will run before continuing! See https://support.zadarastorage.com/hc/en-us/articles/213024926-How-to-enable-IPsec-on-Linux-instances for additional information.

4.2.2 Create the Zadara NFS Mount Configuration By default, with Linux systemd, the IPsec service may stop before the Zadara NFS storage can be completely dismounted. This will cause time-outs for several minutes while trying to stop the VM. To ensure the NFS dismount occurs before IPsec is stopped on a more consistent basis, an NFS mount service must be created. Note: If IPsec is not being used, see section 4.2.3 for mounting the Zadara Storage via fstab.

• Create /usr/lib/systemd/system/mnt-nfs.mount. Note: The following example shows 172.34.140.200:/export/emsnfs as the NFS Server, and /mnt/nfs as the mount point. Change accordingly. If /mnt/nfs is not the mount point, then the


name of the mount service must also be changed. For example, if the mount point is /opt/gcp, then the name of the mount file would be opt-gcp.mount. DO NOT modify the options shown below!

[Unit]

Description=Zadara NFS /mnt/nfs Requires=ipsec.service After=ipsec.service

[Mount]

What=172.34.140.200:/export/emsnfs/ Where=/mnt/nfs Type=nfs

Options=nfsvers=4.0,soft,retrans=2,actimeo=1,timeo=150,_netdev [Install]

WantedBy=multi-user.target

Figure 7 - mnt-nfs.mount service example

• Enable the new mount service to run on startup: sudo systemctl enable mnt-nfs.mount • Create a new mount point for the storage: sudo mkdir /mnt/nfs • Mount the shared storage using sudo systemctl start mnt-nfs.mount

4.2.3 NFS Mount via Fstab If IPsec is not being used, creating a mount service is not required, and the Zadara Storage can be mounted via /etc/fstab. The following example is an example of the Zadara Storage being installed via fstab. As with the mount service, DO NOT modify the options!

172.34.140.200:/export/emsnfs/ /mnt/nfs nfs nfsvers=4.0,soft,retrans=2,actimeo=1,timeo=150,_netdev

Figure 8 - Fstab example

• Create a new mount point for the storage: sudo mkdir /mnt/nfs • Mount the shared storage using sudo mount /mnt/nfs

4.2.4 Linux Kernel Changes The Linux kernel by default can keep the tcp_keepalives for up to twenty minutes. This can have a delayed affect on EMS fail-over. To shorten this time, the Linux kernel property tcp_retries2 can be modified. To modify this property, do the following:

• Edit /etc/sysctl.conf, and add the following value: net.ipv4.tcp_retries2 = 4 • Following is an example of /etc/sysctl.conf with the change:

sysctl settings are defined through files in # /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/. #


# Vendors settings live in /usr/lib/sysctl.d/. # To override a whole file, create a new file with the same in # /etc/sysctl.d/ and put new settings there. To override # only specific settings, add a file with a lexically later # name in /etc/sysctl.d/ and put new settings there. # # For more information, see sysctl.conf(5) and sysctl.d(5). # Update to shorten tcp keep alives net.ipv4.tcp_retries2 = 4

Figure 9 - Sysctl change

• Reboot the GCP instance. After the reboot, check: o The shared file system has mounted o That tcp_retries2 =4. Use cat /proc/sys/net/ipv4/tcp_retries2 to verify

Note: Resolve any issues before continuing!


5 Enterprise Message Service Installation and Configuration

This section will outline the installation and configuration of EMS on the GCP VM instances.

5.1 EMS Installation

EMS 8.5 was used for all testing, and is recommended. Either the Enterprise or Community edition of EMS may be used. Install EMS on all of the of the GCP VM instances. Nothing specific or custom is required to the base configuration of EMS. Follow the EMS 8.5 Installation guide for installing EMS.

Once EMS is installed, use the following to configure EMS on all three VMs.

• EMS 8.5 may have been installed as the root user. If EMS was installed as the root user, change the ownership to the username created by GCP, and used to login into the GCP VM instances.

• Use sudo chown –R <username>:<username> /opt/tibco to change the ownership.

On the EMS server VMs:

• Create /opt/tibco/ems/8.5/bin/logs directory

On one of the EMS server VMs:

• Create the configuration directory on the shared storage:

mkdir –parents /mnt/nfs/tibco/cfgmgmt/ems/data/datastore

• Copy /opt/tibco/ems/8.5/samples/config/*.conf /mnt/nfs/tibco/cfgmgmt/ems/data/

5.2 EMS Configuration

There are specific configuration changes which must be made to provide better write performance and reliability of EMS F/T. This section will discuss these changes. See the EMS User Guide for additional information on setting or the use of, any properties discussed. Note: EMS 8.5 no longer pre-configures the main tibemsd.conf configuration file as part of the installation. All configuration modifications discussed must be completed. Copy tibemsd.conf to tibemsd1.conf and tibemsd2.conf.

5.2.1 Stores.conf In stores.conf, modify/add the following:

The file_minimum=xxGB can be added to each data store. By adding this property, EMS will pre-allocate the space on the shared storage the data store. Note: This is an optional setting, and is not required, but will enhance message write throughput on disk. The minimum should be at least 1GB. Expect the initial startup of EMS to take longer as it creates and allocates the space for the store file.


The file_crc=true should be added, if not already present. This enables EMS to check for data integrity of the data store. This is the default, but ensures it is set.

The following is an example of stores.conf with the changes.

[$sys.failsafe] type=file file=sync-msgs.db mode=sync file_minimum=2GB file_crc=true [sync2] type=file file=sync2-msgs.db mode=sync file_crc=true

Figure 10 - Stores.conf example

5.2.2 Factories.conf The EMS client reconnect properties must be set to enable the EMS client to reconnect to the EMS server in the event of an EMS server failure in an F/T configuration. The reconnect properties can be defined in a number of ways, including in the Java/C code, TIBCO application’s configuration file, and/or through the connection factory when they are used. The default values may be too low in GCP to reliably allow the EMS client to reconnect to the EMS server after a fail-over, especially with network or system failure. It is recommended to set the reconnect_attempt_count to 100, and the reconnect_attempt_delay to 5000. With these values, the EMS client will attempt to reconnect 100 times, every 5 seconds.

The following example shows the values for the FTConnectionFactory in factories.conf. Note: In the following example for the url, <server1> is 10.150.0.5 and the <port1> is 7224, and <server2> is 10.150.0.3 and the <port2> is 7224. Substitute with the appropriate values for the environment.

[FTConnectionFactory] type = generic url = tcp://10.150.0.5:7224,tcp://10.150.0.3:7224 reconnect_attempt_count=100 reconnect_attempt_delay=5000

Figure 11 - Factories Configuration Example

5.2.3 Tibemsd.conf The tibemsd.conf for both EMS Servers needs to be updated for multiple properties.

These include:

• Location of all configuration files – The location must be on the shared storage device.

######################################################################## # Configuration files. ########################################################################


users = "/mnt/nfs/tibco/cfgmgmt/ems/data/users.conf" groups = "/mnt/nfs/tibco/cfgmgmt/ems/data/groups.conf" topics = "/mnt/nfs/tibco/cfgmgmt/ems/data/topics.conf" queues = "/mnt/nfs/tibco/cfgmgmt/ems/data/queues.conf" acl_list = "/mnt/nfs/tibco/cfgmgmt/ems/data/acl.conf" factories = "/mnt/nfs/tibco/cfgmgmt/ems/data/factories.conf" routes = "/mnt/nfs/tibco/cfgmgmt/ems/data/routes.conf" bridges = "/mnt/nfs/tibco/cfgmgmt/ems/data/bridges.conf" transports = "/mnt/nfs/tibco/cfgmgmt/ems/data/transports.conf" tibrvcm = "/mnt/nfs/tibco/cfgmgmt/ems/data/tibrvcm.conf" durables = "/mnt/nfs/tibco/cfgmgmt/ems/data/durables.conf" channels = "/mnt/nfs/tibco/cfgmgmt/ems/data/channels.conf" stores = "/mnt/nfs/tibco/cfgmgmt/ems/data/stores.conf" ######################################################################## # Persistent Storage.

# # store: directory to store persistent messages. ######################################################################## store = "/mnt/nfs/tibco/cfgmgmt/ems/data/datastore"

• Log File location – The location must be on the local disk if the GCP VM instance. The following example locates in the /opt/tibco/ems/8.5/bin/logs directory.

logfile = "/opt/tibco/ems/8.5/bin/logs/tibemsd1.log"

Server and Client Heartbeat and timeout values – These properties determine how long the client/server listen for the heartbeat from the the client/server, before disconnecting. The values shown below have been tested, and work well on GCP with the Zadara NFS storage. The connection_timeout can be increased if necessary, but the heartbeat must remain at 5.

server_heartbeat_client = 5 server_timeout_client_connection = 30 client_heartbeat_server = 5 client_timeout_server_connection = 30

Enabling exiting disk error property – New property since EMS version 8.4. This property defines to EMS to exit when there is a disk error reading/writing to a device. It must be enabled. This property will help prevent “Dual Active Server” conditions, sometimes seen in networked storage devices.

always_exit_on_disk_error = enable

FT properties – Normal properties for the defining the peer EMS server instance, heartbeat between instances, and etc. must be set. Ensure the ft_reconnect_timeout is set to the value shown.

ft_reconnect_timeout = 90 Increase the Maximum Message Memory to 1024 or greater, depending on how much RAM is allocated to the GCP VM.


max_msg_memory = 1024

Define a value for destination_backlog_swapout. This will help limit excessive reads to the shared disk. A minimum of 10000 is recommended. If the queues, will persistent a larger number of messages, increase the size.

destination_backlog_swapout = 10000

5.2.4 Starting the EMS Instances Once the configuration files are updated, EMS can be started. The –forceStart parameter is optional. Start both instances, taking note of which instance is the active EMS instance. Note: It will take ~ 5 minutes for the first EMS instance to become active due to the creation of the data stores on the shared storage. Note: Leave the both windows to the EMS server instances open. This will be needed for the testing.

Figure 12 - Starting an EMS Instance Example


6 Testing EMS Fault Tolerance on GCP with Zadara Storage

Once EMS has been started on both of the GCP VM instances, the failover testing can be performed. This section will outline several test cases, including EMS Server process failure, machine failure, and network failure. Tests are performed using queues with persistence set. This guarantees that the shared file system will be accessed during the tests.

6.1 EMS Client Application Setup

The third GCP VM instance is used to run the test application. EMS is shipped with sample Java applications which can be used for the testing. The tibjmsMsgProducerPerf utility should be used for the testing. All sample Java applications are located in /opt/tibco/ems/8.5/samples/java. Use the following to setup the environment:

• Ensure the version Java 1.8 development environment is installed. • Install EMS version 8.5 on the third GCP VM instance following the EMS installation

procedures. • After the installation of EMS is completed:

o cd to $TIBCO_HOME/ems/8.5/samples/java o . ./setup.sh o javac *.java – This should compile all Java apps in the directory

• Ensure that at least one of the EMS server instances is running (both should be running) • Use the TIBCO EMS Administration Tool to create the EMS Queue sync utilizing the

$sys.failsafe data store. This is required for testing with a synchronous data store: o cd to $TIBCO_HOME/ems/8.5/bin o ./tibemsadmin tcp://<server>:port – of the active EMS instance

Note: In the following examples, <server> is the IP address is 10.150.0.5 and the <port> is 7224. Substitute with the appropriate values for the environment.


Figure 13 - Create the Sync Queue

6.2 Performing the EMS Fault Tolerant Test Cases

Three different tests should be performed: 1. EMS Process failure – Active EMS instance is stopped 2. Network failure – Network failure between the Active EMS Server machine, and the

Zadara VPSA 3. System failure – Accidental restart of the GCP VM running the Active EMS server

instance

This section will outline how to run these three tests, and what the expected results should be. Note: All test cases must be run from the third virtual machine where the Java sample app was compiled.

6.2.1 EMS Process Failure Test This test verifies that an EMS client continues to function correctly, with no message loss during an EMS server process failover. Two EMS server instances will be running in a F/T configuration, while messages are being sent. The active EMS instance will be stopped, the stand-by EMS instance should take over, and continue processing messages until the EMS Java completes publishing messages. Note: In the following examples, <server1> the IP address is 10.150.0.5 and the <port1> is 7224, and <server2> the IP address is 10.150.0.3 and the <port2> is 7224. Substitute with the appropriate values for the environment.


6.2.1.1 Running the Process Failure Test Three ssh terminal sessions are needed for this test; one for server1, one for server2, and one for the client Start EMS on server1 and server2 in the foreground. EMS on server1 should be the active EMS instance. From the client, start the Java application

o cd to $TIBCO_HOME/ems/8.5/samples/java o . ./setup.sh o java tibjmsMsgProducerPerf –server tcp://10.150.0.5:7224, tcp://10.150.0.3:7224 –

factory FTConnectionFactory –delivery PERSISTENT –connections 10 –threads 8 –count 20000 –size 1024 –queue sync

Figure 14 - Running the tibjmsMsgProducerPerf Application

Immediately kill/stop the EMS instance on server1, with cntrl-C The standby EMS instance on server2 will become active, and recover all messages. It should be possible to stop and start the EMS instances a few times while the Java test application is running. The number of recovered messages will increase.


Figure 15 - The Standby EMS Instance becoming active on Server2

After the Java application completes, run tibemsadmin tcp://10.150.0.3:7224 (or from server1, if it is active), to verify that there are 20000 messages in the sync queue. Restart the EMS instance on server1, and stop the EMS instance on server2. EMS on server1 should become active, and recover all 20K messages with no errors. Use tibemsadmin and purge the sync queue in preparation for the next test.


Figure 16 - Purge the Sync queue for tibemsadmin

• Stop and restart EMS on server1 and server2 in the foreground. EMS on server1 should be the active EMS instance.

6.2.1.2 Expected Results The Java test application should complete, with a slight pause during failover but should resume sending messages once the failover is complete. No messages should be lost. There should not be more than 20K messages, and there never should there be less than 20K. Depending on the number of messages that must be recovered, the fail-over should be very short, possibly less than 1 second. Complete fail-over from one server to the other with the 20K messages should be under 10 seconds.

6.2.2 Network Failure Test This test verifies that an EMS client continues to function correctly, with no message loss during a network failure between the active EMS server instance, and the Zadara VPSA. Two EMS server instances will be running in a F/T configuration, while messages are being sent. The TCP/NFS port will be blocked between the active EMS instance and the Zadara VPSA via iptables. The active EMS instance should get a write error, and exit, allowing the stand-by EMS instance to gain the locks on the EMS data stores, and take over. The EMS Java application should continue processing messages until it completes. Note: In the following examples, <server1> IP address is 10.150.0.5 and the <port1> is 7224, and <server2> IP address 10.150.0.3 and the <port2> is 7224. Substitute with the appropriate values for the environment.


6.2.2.1 Running the Network Failure Test Four ssh terminal sessions are needed for this test; two for server1, one for server2, and one for the client A script will be needed to block the NFS port (2049) on server1 while the Java app is publishing messages. The following figure shows the drop_nfs.sh script. Cut and past the following to create the script. The script must be created in the second ssh terminal window on server1.

# # Script to get the current iptables definitions, drop NFS port, then restore the original table defi #nitions. # echo " Saving existing IP table definitions" echo "" sudo iptables-save >iptables_save # # Drop the NFS port 2049 # date echo " Dropping NFS 2049 port" echo "" sudo iptables -A INPUT -p TCP --dport 2049 -j DROP sudo iptables -A OUTPUT -p TCP --dport 2049 -j DROP # # Sleep for 3 minutes # echo " Sleeping for 3 minutes..." echo "" sleep 3m # # Restore original IP table definitions" # echo " Restoring original IP table definitions" echo "" sudo iptables -F sudo iptables-restore <iptables_save # date echo "Done."

Figure 17 - drop_nfs.sh script

From the client, start the Java application o cd to $TIBCO_HOME/ems/8.5/samples/java o . ./setup.sh o java tibjmsMsgProducerPerf –server tcp://10.150.0.5:7224, tcp://10.150.0.3:7224 –


From the second ssh terminal window on server1, run drop_nfs.sh


Figure 18 - Running drop_nfs.sh script

• The active EMS instance on server1 should terminate with a disk write error:

Figure 19 - Disk Write Error on Server1

The standby EMS instance on server2 should determine EMS on server1 is no longer producing a heartbeat, will attempt to become active. This should only take a few seconds for server2 to become active, depending on the amount of data to be recovered. There may be other warnings, depending on how long it takes for server2 to obtain the locks.


Figure 20 - Server2 becoming Active after network failure

After the Java application completes, run tibemsadmin tcp://10.150.0.3:7224 (or on server1, if it is active), to verify that there is a minimum of 20000 messages in the sync queue. Restart the EMS instance on server1, and stop the EMS instance on server2. EMS on server1 should become active, and recover all 20K messages with no errors. While still in tibemsadmin, purge the sync queue in preparation for the next test. Stop and restart EMS on server1 and server2 in the foreground. EMS on server1 should be the active EMS instance.

6.2.2.2 Expected Results The Java test application should complete, pausing during the failover, but should resume sending messages once the failover is complete. No messages should be lost. There can be more than 20K messages, depending on the number of connections/threads, but there should never be less than 20K. Depending on the number of messages that must be recovered, the fail-over should only take a few seconds. Note: In some circumstances, the following error may occur with the network failure, if a file_minimum for a data store has been set. SEVERE ERROR: Error occurred while processing Message record id=xxxxxx (get record) - Unable to reconstruct message: 74 - IO failed


This can sometimes occur when there is a file_minimum set, and the file is only partially filled. This is not an error, and can be ignored. All messages will be recovered. If this is seen, restart the tibemsd server with the –forceStart option as shown in Figure 12.

6.2.3 System Failure Test This test verifies that an EMS client continues to function correctly, with no message loss during a system failure on the GCP VM instance running the active EMS server instance. This is not a normal occurrence. However, it is possible to accidentally restart the virtual machine from the GCP console. Two EMS server instances will be running in a F/T configuration, while messages are being sent. From the GCP console, the VM instance where the active EMS instance is running will be restarted. The stand-by EMS instance should be able to gain the locks on the EMS data stores, and take over. The EMS Java application should continue processing messages until it completes. Note: In the following examples, <server1> IP address is 10.150.0.5 and the <port1> is 7224, and <server2> IP address is 10.150.0.3 and the <port2> is 7224. Substitute with the appropriate values for the environment.

6.2.3.1 Running the System Failure Test Three ssh terminal sessions are needed for this test; one from each of the virtual machines. The GCP console must also be available, and be on the Compute Engines/VM Instances dashboard page. Note: In the examples below, ems4-3 is server1

Figure 21 - GCP Compute Engine/VM Instances page

• From the client, start the Java application: o cd to $TIBCO_HOME/ems/8.5/samples/java o . ./setup.sh o java tibjmsMsgProducerPerf –server tcp://10.150.0.5:7224, tcp://10.150.0.3:7224 –


• From the GCP dashboard, click on the server1 instance, then click on the reset button, which is next to the trash can. Verify to reset the server. This will reboot the server1 GCP VM instance.


Note/Warning: A reset from the GCP console should not be used other than for this test. In some situations, IPsec will stop before the Zadara storage has completely dismounted, and it may take several minutes for the VM to reset. This should not occur during a normal shutdown/reboot. Additionally, a reset can cause file system corruption, and login issues.

• The ssh terminal window to the server1 GCP VM instance should immediately terminate, and the stand-by EMS instance should recover all messages, and become active within a few seconds, depending on the number of messages to be recovered. There may be other warnings, depending on how long it takes for server2 to obtain the locks.

Figure 22 – Standby EMS instance recovering from system failure of primary

• After the Java application completes, run tibemsadmin tcp://server2:7224 (or to the active EMS instance), to verify that there is a minimum of 20000 messages in the sync queue.

• Restart the EMS instance on the restarted GCP VM instance, and stop the currently active EMS instance on the second GCP VM instance. EMS should become active on the restarted instance, and recover all 20K messages with no errors.

• Use tibemsadmin to verify, then purge the sync queue. • Stop EMS on both GCP VM instances. • This concludes the tests, so all processes, terminal windows, and GCP VM instances can be

stopped.


6.2.3.2 Expected Results The Java test application should complete, pausing during failover, but should resume sending messages once the failover is complete. No messages should be lost. There can be more than 20K messages, depending on the number of connections/threads, but there should never be less than 20K messages. Depending on the number of messages that must be recovered, the fail-over should take less than a minute. It has been observed with the system failover on GCP with the Zadara shared storage, it may take a few minutes to release the locks, and allow the standby EMS instance to become active. There may also be additional warning regarding the clients trying to reconnect. Note: In some circumstances, the following error may occur with an abrupt system shutdown, if a file_minimum for a data store has been set. SEVERE ERROR: Error occurred while processing Message record id=xxxxxx (get record) - Unable to reconstruct message: 74 - IO failed

This can sometimes occur when there is a file_minimum set, and the file is only partially filled. This is not an error, and can be ignored. All messages will be recovered. If this is seen, restart the tibemsd server with the –forceStart option as shown in Figure 12.

configuring a tibco enterprise message service environment ......• setting up linux vm instance on...

Documents