cloud rin installation guide · 12/7/2020 · the rin can be installed using the command line or...

SNYPR 6.3.1

CloudRIN Installation Guide

Date Published: 3/22/2021

Securonix Proprietary Statement

This material constitutes proprietary and trade secret information of Securonix, and shall not be disclosed to any

third party, nor used by the recipient except under the terms and conditions prescribed by Securonix.

The trademarks, service marks, and logos of Securonix and others used herein are the property of Securonix or their

respective owners.

Securonix Copyright Statement

This material is also protected by Federal Copyright Law and is not to be copied or reproduced in any form, using any

medium, without the prior written authorization of Securonix.

However, Securonix allows the printing of the Adobe Acrobat PDF files for the purposes of client training and

reference.

Information in this document is subject to change without notice. The software described in this document is

furnished under a license agreement or nondisclosure agreement. The software may be used or copied only in

accordance with the terms of those agreements. Nothing herein should be construed as constituting an additional

warranty. Securonix shall not be liable for technical or editorial errors or omissions contained herein. No part of this

publication may be reproduced, stored in a retrieval system, or transmitted in any form or any means electronic or

mechanical, including photocopying and recording for any purpose other than the purchaser's internal use without

the written permission of Securonix.

Copyright © 2021 Securonix. All rights reserved.

Contact Information

Securonix

5080 Spectrum Drive, Suite 950W

Addison, TX 75001

(855) 732-6649

SNYPR Remote Ingestion Guide 2

Table of ContentsIntroduction 4

Server Recommendation 4Prerequisites 5

Remote Ingester Node Installation 8

Upgrade to RIN 6.3.1 19

Remote Management for RIN 29

Communication Flow 29Manage RIN in SNYPR 30Configure Syslog Filters() and Source() in Activity Import 33

Network Performance Tuning 37

Reference Server Configuration 37RIN Tuning Process 38Best Practices for Network Tuning 47Troubleshooting Common Errors 49

Uninstall the RIN 50

Troubleshoot the RIN 55

Appendix A: Configure Proxy Setup 69

Sample HTTP and HTTPS Settings 69


Introduction

IntroductionThe Remote Ingestion Node (RIN) is a lightweight Java program that is used to forward logs, in real-time, from a remote server to the SNYPR ingestion nodes (also referred to as Kafka Brokers). Forwarding logs in real-time provides the ability to ingest and analyze events as soon as they are generated with a minimum delay.

The RIN offers you the following advantages:

l Forwards logs from various data centers and locations

l Compresses the data to reduce network bandwidth utilization

l Encrypts data to secure the communications

l Maintains a local cache and retransmits data in case of communications failure

Server RecommendationThe following table describes the RIN sizing recommendations for a small, medium and large configuration:

Recommendation Minimal (

Introduction

Note: For a system to support large number of TCP connections, irrespective of the

EPS, 10 GB NIC is required.

Prerequisites You have to ensure that the following prerequisites are available before you install the RIN:

l Server Requirements: The RIN servers can be physical servers or virtual machines.

l Firewall Configuration: The firewall ports must be open for the RIN server to com-municate with SNYPR.

l General Requirement: The network validation utility must be installed and SELinux must be permissive.

l Root Permission: The RIN installation requires a non-root user account with sudo permission to install the Linux services.

The following table describes the prerequisites:

Requirement

TypeRequirement Description

Server Operating System CentOS 7 or Red Hat 7.x

Server Data Retention on the RIN

4 days

Firewall Ports SNYPR ConsoleOutbound port 80, 443, 8080, and 8443 must be open.

Firewall Ports KAFKA Brokers Outbound port 9092 or 9093 must be open.


Introduction

Requirement


Firewall Ports RIN Syslog sources

Inbound port 514 must be open.

Note: Use TCP for Syslog sources to

improve the reliability of data

transfer.

General Network Validation Utility

tcptraceroute must be installed.

General SELinux Module

The SELinux module must be in the permissive

mode.

You can run getenforce as the root user to

check the status of SELinux.

If the status is not permissive, you can run

setenforce 0 as the root user to change it

to permissive.


Introduction

Requirement


General Root Permission

You must have root permission to assign sudo

permission to a non-root user account.

To add a non-root user, such as securonix, with

sudo permission, enter the following command

in the Terminal application:

useradd securonix

passwd securonix

sudo usermod -aG wheel securonix


Remote Ingester Node Installation

Remote Ingester Node InstallationThis section describes how you can install and configure the RIN for data collection. The RIN can be installed using the command line or Graphical User Interface (GUI).

Step 1: Pre Installation Setup

This section describes the following steps and checks that you must perform before installing RIN:

l Create a Securonix directory.

l Download the RIN Installer package.

l Run the prerequisite validation script to check if all the prerequisites are available.

Note: The IP and Hostname cannot be changed after the RIN installation.

Complete the following pre-installation tasks to prepare for the installation:

1. Access the Terminal application from your Linux server.

2. Create a directory with name as Securonix under the / (root) directory using the following command:

sudo mkdir /Securonix

This step is required as syslog is deployed in the Securonix directory under the / root directory.



Note: Ensure that the Securonix directory is created by the user who has sudo

permission and is going to install RIN. For more information, see Root

Permission.

3. Perform the following steps if you are not logged in using the securonix account:

a. Assign ownership to the securonix account.

sudo chown -R securonix:securonix /Securonix

b. Log in as the securonix user.

su - securonix

c. Enter the password for the securonix account.

When you successfully login as the securonix user, securonix is displayed in the command prompt as highlighted below.

4. Run the following command to change the directory to /Securonix, if needed:



cd /Securonix

5. Download the RIN package and copy it in the /Securonix directory created in Step2.

For example: If you can are copying the RIN package from a system with Mac OS, then you will use SCP command.

scp SNYPR-RIN-{tenantname}.tar username@{IPaddress}:/username

If you are copying the RIN package from a system with Windows OS, then you have to use file transfer tools such as WINSCP.

The RIN package is in the tar format.

6. Untar (extract) the RIN package in the /Securonix directory using the following command:

tar xvf SNYPR-RIN-{tenantname}.tar

Note: The files are extracted in the RIN folder available at the following

location: /Securonix/SNYPR-RIN-{tenantname}/{tenantname}/. You have to

navigate this folder to run the validation script.

7. Run the following command to view the files available in the /Securonix directory:

ls /Securonix



The SNYPR-RIN-{tenantname} folder has the extracted RIN installation files.

8. Run the following command to change the directory to SNYPR-RIN-{tenantname}:

cd SNYPR-RIN-{tenantname}

The directory has changed to SNYPR-RIN-{tenantname} in the following screen:

9. Run the following command to view the files available in the /Securonix/SNYPR-RIN-{tenantname} folder:

ls

The SNYPR-RIN-{tenantname} folder is available:



10. Run the following command to change the directory to {tenantname}/RIN:

cd {tenantname}/RIN

11. Run the following command to view the files available in the RIN folder:

ls

The following files and folders are available inside the RIN folder:

12. Run the validation scrips to check if you have all the prerequisites available for RIN installation, using the following command:



sh validation.sh pre-check

Note: Ensure that you are running the command for RIN directory.

If prerequisites are not installed or available, the system generates an error. Refer to RIN Pre Installation Issues for information on error codes.

Step 2: Prep Installation Setup

This section describes the following steps and checks that you must perform before installing RIN:

l Validate the connectivity with SNYPR.

l Specify the RIN installation location.

Complete the following prep-installation tasks to prepare for the installation:

1. Ensure that you are in the following folder path in Terminal: /Securonix/SNYPR-RIN-{tenantname}/{tenantname}/RIN.

Note: You can run the pwd command to validate your current folder location.

2. Run the following command in Terminal:

sh validation.sh prepare-to-install



The command prompt displaysChoose Install Folde.

3. Type /Securonix and press Enter. The command prompt displays the confirmation message.

4. Type Y. The screen displays the Enter SUDO Password section.

5. Enter the non-root password of the securonix account and press Enter.

The system validates the connection details. If the connection is unsuccessful, the system generates an error. Refer to RIN Prep Installation Issues for information on error codes.

Step 3: RIN Installation

1. Run the following command to change the directory to /RIN, if needed:

cd /Securonix/SNYPR-RIN-{tenantName}/{tenantname}/RIN

2. Run the following command to launch the installer:

./.bin

The ingester installation file has .bin as suffix.

Note: You must install RIN as a non-root user with the sudo permission.

3. Press Enter. The installation starts.



Note: The SNYPR Console must be running and accessible on the network from

the server where the RIN is installed.

The command prompt displays the "The Installation of Ingester is complete" message once the installation is complete.

If the steps fail, refer to the Troubleshooting section for information on error codes.

Note: You can refer to the Readme files located at /README file for information on post installation steps. You can also

refer to installer logs from /SNYPR_install-

ation/Logs/.

Step 4: Post Installation

Check the network connectivity by running the following command from Terminal:

1. Run the following command:

source /home/securonix/.bash_profile

2. Run the following command for post validation:

sh validation.sh post-check



3. Start the gateway with the following command:

sudo systemctl start scnx-gateway

4. Check the gateway status with the following command to confirm that it has started:

sudo systemctl status scnx-gateway

5. Start the Remote Ingester as a securonix user with the following command:

sudo systemctl start scnx-ingester

6. Check the Remote Ingester status with the following command to confirm that it

has started:

sudo systemctl status scnx-ingester

If there is an error or you want to check the Remote Ingester logs, use this

command:

tail -1234f //logs/Ingester.log

7. Start the Syslog server as securonix user with the following command:

sudo systemctl start scnx-syslog-ng

8. Check the status of the Syslog server with the following command:



sudo systemctl status scnx-syslog-ng

You can check the logs for the Syslog server by using the following command:

journalctl -f -u scnx-syslog-ng -n 1234

Note: Ensure the directory is set to /Securonix before running the command.

If connection is unsuccessful, the system generates an error. Refer to the Troubleshooting section for information on error codes.

Note: If there is a proxy server configured between RIN and SNYPR application,

then you have to perform some additional settings. For information on proxy server

setup, refer to the Appendix A: Configure Proxy Setup section.

Step 5: Verify the RIN Connectivity to Console

When the RIN starts, it validates the token with the application. If the connection is successful, the Token Validated message is displayed. If the connection fails, the

Remote Ingester shuts down.

You can verify the RIN connectivity from the Menu > Administrator > Settings > Manage Ingesters.


Upgrade to RIN 6.3.1

Upgrade to RIN 6.3.1This section describes how you can upgrade and configure the RIN for data collection.

Step 1: Pre Upgrade Setup

This section describes the following steps that you must perform before upgrading RIN:

l Download the RIN Installer package.

l Run the prerequisite validation script to check if all the prerequisites are available. This step is optional.

Complete the following tasks to prepare for the upgrade:


Note: Ensure you have logged in using the user account you used to install the

earlier version of RIN.

2. Run the following command to change the directory to /Securonix if you have installed the previous version of RIN in this folder:

cd /Securonix

3. Download the RIN package and copy it in the /Securonix directory. The RIN package is in the tgz format.

4. Untar (extract) the RIN package in the /Securonix directory using the following



command:

tar xvf SNYPR-RIN-{tenantname}.tgz

Note: The files are extracted in the RIN folder available at the following

location: /Securonix/SNYPR-RIN-{tenantname}/{tenantname}/. You have to

navigate this folder to run the validation script.

5. Run the following command to view the files available in the /Securonix directory:

ls /Securonix

The folder has the extracted RIN installation files.

6. Run the following command to navigate to the /Securonix/SNYPR-RIN-{tenantname}/{tenantname}/RIN

cd /Securonix/SNYPR-RIN-{tenantname}/{tenantname}/RIN

7. Run the following command to view the files available in the RIN folder:

ls



The following files and folders are available inside the RIN folder:

8. Run validation script to check if all the prerequisites are available for RIN upgrade. This is an optional step.

sh validation.sh pre-check

If prerequisites are not available, the system generates an error. Refer to RIN Pre Installation Issues for information on error codes.

Step 2: Prep Upgrade Setup

This section describes the following steps and checks that you must perform before upgrading RIN:

l Set the connection details in the installer.properties file.

l Validate the connectivity with SNYPR.

Complete the following tasks to prepare for the upgrade:

1. Ensure that you are in the following folder path in Terminal: /Securonix/SNYPR-RIN-{tenantname}/{tenantname}/RIN.



Note: You can run the pwd command to validate your current folder location.

2. Open installer.properties using the following command:

vi installer.properties

The installer.properties file's parameters are displayed in Terminal.



3. Press i on your keyboard to display the Terminal application in the Insert mode.

The Terminal application displays "Insert" when the mode is enabled.



4. Enter or verify the values for the following settings in installer.properties:

Setting Type Setting Description

Choose Deployment Mode

DEPLOY_MODEDisplays whether it is a new installation or upgrade.

Choose Install Folder

USER_INSTALL_DIR

Enter or verify the directory name as /Securonix. This is the directory that you created in the Pre-Installation section.




Enter SUDO Password

SYSVAR_PWDEnter the sudo password, the non-root password of the securonix account.

Get Snypr Application Details

T_URL

Enter or verify the SNYPR application

URL. For example, https://FQDN/Snypr.

This URL is provided in the "RIN

Installation" email sent by the Securonix

Onboarding team.


T_USERNAME

Enter or verify the admin user name of

the SNYPR application.

The admin user name is provided in the

"RIN Installation" email sent by the

Securonix Onboarding team.


T_PASSWORDEnter password for the specified user name.


T_Tenant

Enter or verify the tenant name.

The tenant name is provided in the "RIN


Onboarding team.


Tnt_ID

Enter or verify the tenant ID.

The tenant ID is provided in the "RIN


Onboarding team.

Get Tenant Type Tenant_Type Enter or verify Isolated as tenant type.




Get MSSP Type MSSP_TYPE

Do not enter any value for this setting.

This setting is only applicable when

Tenant_Type is MSSP.

5. Press Esc to exit from the Insert mode.

6. Type :wq! to save the changes.


sh validation.sh prepare-to-install

The system validates the connection details. If connection is unsuccessful, the system generates an error. Refer to RIN Prep Installation Issues for information on error codes.

Step 3: RIN Upgrade

1. Run the following command to change the directory to /RIN, if needed:

cd /Securonix/SNYPR-RIN-{tenantName}/{tenantname}/RIN



2. Run the following command to launch the installer:

./.bin

The ingester installation file has .bin as suffix.

3. Press Enter. The upgrade starts.

Note: The SNYPR Console must be running and accessible on the network from

the server where the RIN is installed.

The command prompt displays the "The Installation of Ingester is complete" message after the upgrade is complete.

If the steps fail, refer to RIN Post Installation Issues for information on error codes.

Note: You can refer to the Readme files located at /README file for information on post upgrade steps. You can also refer

to installer logs from /SNYPR_installation/Logs/.

Step 4: Post Upgrade

Check the network connectivity by running the following command from Terminal:

sh validation.sh post-check

If connection is unsuccessful, the system generates an error. Refer to RIN Post Installation Issues for information on error codes



Note: Ensure the directory is set to /Securonix before running the command.

Step 5: Verify the RIN Connectivity to Console

When the RIN starts, it validates the token with the application. If the connection is successful, the Token Validated message is displayed. If the connection fails, the Remote Ingester shuts down.

You can verify the RIN connectivity from the Menu > Administrator > Settings > Manage Ingesters.


Remote Management for RIN

Remote Management for RINSNYPR provides remote management for the RIN with the new Gateway.

SNYPR Gateway

SNYPR Gateway is a remote access solution that enables you to control your application servers from anywhere in the world. It provides the ability to take selective actions on the edge-node applications used by SNYPR, directly from the console. This allows unified management and running of all SNYPR applications and services, as well as simplify maintenance of these services to enhance end-user experience.

SNYPR Gateway for the RIN provides the ability to start, stop, and restart the RIN from any location directly from SNYPR. This also allows you to access and download RIN logs so you can quickly collect logs for troubleshooting.

This section covers the following topics:

l Communication Flow

l Manage RIN in SNYPR

l Configure Syslog Filters() and Source() in Activity Import

Communication FlowThe diagram below shows how the SNYPR UI uses Gateway to communicate with the RIN:



When you install the RIN upgrade for SNYPR CU4, these two new Kafka topics are created.

Manage RIN in SNYPRIn SNYPR, you can manage the status of one or multiple RINs using the Manage Ingesters screen. This screen shows individual ingester details, ingester status, and available actions per ingester.

To access this screen, navigate to Menu > Administration > Settings.



Once ingester is registered in the SNYPRUI, it displays under the Manage Ingesters screen. Each ingester has a color code that reflects the status of the ingester. The color codes are:

l Green: The application is successfully connected to the ingester and is running.

l Yellow: The ingester status is refreshing.

l Red: The application failed to connect to the Ingester and has stopped.

These colors also display in the Ingesters header, which shows a quick view of the total, running, and stopped ingesters on the screen.

Understanding the Action IconsThe action icons on the Manage Ingesters screen are used to perform specific activities. The table below lists the action icons along with a brief description of its purpose:

Icon Name Description

Stop Stops the individual ingester.

Start Starts the individual ingester.

Restart Restarts the ingester.

DownloadDownloads the logs for individual ingester.




Additional options

l Create new syslog source ()

l View syslog configuration

Refresh allRefreshes content for all the ingesters on the screen.

Creating a New Syslog Source()You can add or edit a new syslog source() for every ingester. To add a new syslog source for a specific ingester, click the Additional Options icon, then select Create/Edit Source.

A list of existing sources will display for the ingester. From here, you can:

l Add, edit, or delete a syslog-ng source () block

l Configure multiple sources ()

Click + Create new towards the bottom of the screen to add a syslog-ng source block. An Add new source section will display:



On the Add new source screen you will specify the details for the new source. The Source name is used as an identifier in the source statement of the syslog-ng configuration file to receive log messages. The Source expression is used to build the source statement.

Once you have completed the fields in this section, click Create.

Configure Syslog Filters() and Source() in Activity ImportYou can add / edit syslog filters per ingester for each datasource directly from the Activity Import screen. You also have the ability to select multiple syslog sources () for each datasource from the same screen.

To access this screen, navigate to Menu > Add Data > Activity. Click + > Add Data for Existing Device Type. For more information on how to add a datasource, see the Activity Data section in the Integration Guide.



To configure a syslog filter, click + in the filters section. An Add New filter pop-up will display where you will specify the details for your filter. Once you have completed all the information in the pop-up, click Add.

The action icons on the Activity Import screen are used to perform multiple activities. The table below lists the action icons along with a brief description of its purpose:


AddAdds a new syslog filter per datasource.

Add Ingester Adds an ingester.

CollapseCollapses all configurations.




Settings

There are three

options when you

click this icon,

including:

l Validate ingester: This option allows you to validate configurations for one or multiple ingesters.

l View logs: This option lets you view logs for the

ingester.

l Remove: This option lets you remove an ingester.

The changes you make are written to a specific, separate section in the syslogng.conf file that have been reserved for sources() and syslog filters() configured from the UI. Do not make changes to this section of the syslogng.conf file from the back-end

Any change made from the UI will override the changes made from the back-end.



Ingesting Data from Multiple RINs You can ingest data coming from multiple RINs into the same datasource. For example, you can have Windows data from multiple data-centers and geographical locations, and ingest the data as part of the same datasource, so you can efficiently search and analyze related events.

To ingest data from multiple ingesters as part of the same datasource, the log format must be the same. If you have two different log formats for Windows, you must create two different datasources.

You also have the ability to add as many ingesters as needed. Once configured, you can validate the configurations for one or multiple ingesters.

The diagram below provides an overview of how multiple RINs (RIN1, RIN2, RIN3) are processed to the same datasource (RG1 and RG2):

Data for a single datasource must be published to the same Kafka RAW Topic from all the RINs as seen in the image above.


Network Performance Tuning

Network Performance Tuning This topic explains how to tune your network for improved performance. You can perform network performance tuning at the time of RIN installation.

The configuration has been tested to support TCP connections for 3K - 5K hosts providing a continuous stream of data. The NIC on the server is 10GB to support the increased loads.

When data is forwarded from a SIEM to RIN, the number of TCP connections established are minimum (less than 50). In this scenario, high number of connections are not a bottleneck and aggressive tuning is not suggested. For high Events Per Second (EPS) environment, dedicated resources are used, for example CPU, RAM, and networking resources when referencing to the Virtual Machine environment.

Reference Server Configuration 1. Run the following command in Terminal to view you server configuration:

lscpu

2. Compare your server details with the following reference server configuration details:

Server Component / Setting Configuration Value

Architecture x86_64

CPU op-mode(s) 32-bit, 64-bit

Byte Order Little Endian

CPU(s) 8

On-line CPU(s) list 0-7



Server Component / Setting Configuration Value

Thread(s) per core 1

Core(s) per socket 2

Socket(s) 4

NUMA node(s) 1

Vendor ID GenuineIntel

CPU family 6

Model 58

Model nameIntel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz

Stepping 0

CPU MHz 2700.000

BogoMIPS 5400.00

Hypervisor vendor VMware

Virtualization type full

L1d cache 32K

L1i cache 32K

L2 cache 256K

L3 cache 30720K

NUMA node0 CPU(s) 0-7

RIN Tuning Process The RIN tuning process consists of the following steps:

Server Preparation / OS Tuning

The server preparation is recommended to support the high EPS tuning. The steps



performed are to tune the OS to support high number of client connections and high EPS.

It is recommended to:

l Add network monitoring tools to gather statistics and debug when errors are present.

l Install netstat, rpm -ivh net-tools-1.60-114.el6.x86_64.rpm.

For more rpm packages, refer to the following:

https://rpmfind.net/linux/rpm2html/search.php?query=%2Fbin%2Fnetstat

l Install ethtool, rpm -ivh ethtool-3.5-6.el6.x86_64.rpm.

For more rpm packages, refer to the following:

http://fr2.rpmfind.net/linux/rpm2html/search.php?query=ethtool

l Use dedicated resources to ensure optimal performance for the collectors.

l Set the latency sensitivity as High for the following scenarios:

l Unfiltered EPS> 10K

l Inbound TCP connections > 10

l Complex Filters

Note: You can set the latency setting from: Edit Settings > VM Options > Latency

Sensitivity.

Server Network Parameters Tuning

1. Edit sysctl.conf file using the following command:



vi /etc/sysctl.conf

2. Add the following parameters to sysctl.conf:



3. Reload the sysctl configurations using the following command:

sysctl -p

4. Increase Transmit Queue Length for 10G NICs.

/sbin/ifconfig

txqueuelen 10000

5. Set the txqueuelen permanently:



vi /etc/rc.local

Syslog-NG Tuning

l High EPS environment tuning: For Improving performance with lots of connections, use the following settings:

l max_connections = active_connections

l log_iw_size = number of active_connections * EPS

l log_fetch_limit = 10000

l flush_lines = 10000

l log_fifo_size = log_iw_size * 10



l LoC EPS environment tuning: For improving performance with a few connections but high amount of traffic, use the following settings:

l log_iw_size = number of active connections * 100,000 or number of active con-nections * EPS whichever is greater

l log_fetch_limit = number of active connections * 100,000 or number of active connections * EPS whichever is greater

l log_fifo_size = log_fifo_size = log_iw_size * 10

l flush_lines = 10,000 or greater



Best Practices for Network TuningData Collection

The fastest way the syslog-ng application can receive log messages from the network is using plain TCP transport with the network source driver. By default, syslog-ng runs



in multi-threaded mode to scale to multiple CPUs or cores for the increased performance.

A TCP-based network source scales based on the number of active connections. This means that if there are 10 incoming TCP connections all coming to the same network source, then that source can use 10 threads, one thread for each connection.

Higher stats_level decreases the performance. For example, stats_level(2) means -10% in performance.

Data Processing and Filtering

Message processors, such as filters, rewrite rules, and parsers, are executed by the reader thread in a sequential manner. Simple filtering (for example, filtering on facility or tag) has no impact on performance at all. However , regular expressions, even simple ones, significantly decrease the message-processing rate by about 40-45%.

It is advised to use the simplest filters when filtering incoming messages. If a message can be filtered with several types of filters, check the measured data. A message when filtered with a regexp , the performance of syslog-ng can drop down to 55-60% of the original performance level. If the tag or facility filters are used, there is no decrease in performance.

When using multiple filters one after the other, or connecting filters with the logical AND/OR operators, the order of filters has a significant impact on performance. You can prioritize filters that are the most likely to match the incoming log messages to the top of the configuration file.

Data Connections

If there are several thousand active connections simultaneously, it is advised to place relay syslog-ng-s on another computer before the syslog-ng server. The volume of



incoming messages is usually not significant but switching between active connections is time-consuming. You can use relays, since they are collecting the logs, to resolve this issue. The syslog-ng solution can easily handle lots of log messages sent from a few connections.

Note: For a system to support large number of TCP connections, irrespective of the

EPS, 10 GB NIC is required. The NIC bonding can be carried out if VM cannot

provide dedicated 10G NIC.

Troubleshooting Common ErrorsThis section explains the common errors during the tuning process:

Error Description & Resolution

Rx DropsSignifies that there is a network issue, for example: faulty

network, faulty cable, or bad interface.

Interface not sending ACK

Implies that there is a contention on the NIC and the NIC is unable to handle the load.

Files not getting created

Signifies that either there is a configuration error in syslog-ng or the environment's file handler limit has been met for the user who is creating the files.


Uninstall the RIN

Uninstall the RINUse the RIN_uninstall.sh script to uninstall RIN.


2. Add the following command to change the directory to /Securonix:

cd /Securonix

3. Add the following command to access the folder where RIN_uninstall.sh is located:

cd Uninstall

4. Add the following command to run RIN_uninstall.sh:

./RIN_uninstall.sh


Uninstall the RIN

5. Type yes for Are you sure you want to process and uninstall Remote Ingester?

and press Enter. The screen displays the Enter SUDO Password section.

6. Enter the password of the securonix account and press Enter. The uninstall process

starts.


Uninstall the RIN

7. Add the following command to access bash_profile:

vi ~/.bash_profile

8. Locate the INGESTER_HOME entry.


Uninstall the RIN

9. Press i on your keyboard to display the Terminal application in the Insert mode.

The Terminal application displays "Insert" when the mode is enabled.


Uninstall the RIN

10. Delete export INGESTER_HOME=/Securonix/Ingester.


12. Type :wq! to save the changes. The RIN is uninstalled.


Troubleshoot the RIN

Troubleshoot the RINThis section highlights some common troubleshooting issues that may appear with the RIN on the SNYPR Console.

RIN Pre Installation Issues

Error CodeValidation

TypeTroubleshooting

RIN-PRE-001

Operating System version

Signifies that the operating system is not correct.

RIN Installer works only with CentOS 7 and Red

Hat 7.x .

RIN-PRE-002Check if running as root user

Signifies that the user is running the pre-

installation steps as a root user.

You have to switch to non-root user account using

the following command:

su

For example :- su securonix




TypeTroubleshooting

RIN-PRE-003Local Firewall configuration

Signifies that ports are not open in firewall.

Check the firewall configuration to ensure that

ports are open.

If you want to turn off the firewall, then use the

following commands:

1. systemctl stop firewalld

2. systemctl disable firewalld

See Firewall Ports for more information.




TypeTroubleshooting

RIN-PRE-004selinux configuration

Signifies that the SELinux module is not in the

permissive mode.

l Option 1: You can run getenforce as the root user to check the status of SELinux. If the

status is not permissive, you can run

setenforce 0 as a root user to change it to

permissive.

You must reboot the server to save the

changes by running sudo reboot.

l Option 2: As a root user execute the following commands:

1. vi /etc/selinux/config

2. SELINUX=permissive

RIN-PRE-005rsyslog disable

As the root user execute the following commands:

1. systemctl stop rsyslog

2. systemctl disable rsysylog




TypeTroubleshooting

RIN-PRE-006

Securonix Directory

Signifies the user does not have permission or

ownership to access the /Securonix directory.

As the root user execute

1. mkdir -p /Securonix

2. chown INSTALLATION_USER:INSTALLATION_USER /Securonix

chmod 775 /Securonix

RIN-PRE-007

disk space

Signifies insufficient disk space. Ensure that you

have provided at least 10GB disk space to

/Securonix.

See Server Recommendation for more

information.

RIN-PRE-008

Source bash profile with user installing the service

Execute bash_profile and validate Ingester_Home

by using the following commands:

source ~/.bash_profile

echo $INGESTER_HOME

When you run the echo command, the command

prompt displays the installation path of the

Ingester.



RIN Prep Installation Issues


TypeTroubleshooting

RIN-PREP-001 Get Install Directory

Signifies either the installation directory is not

created or it does not have the correct ownership.

Ensure that you have specified the correct

directory name that you have created in the Step2,

/Securonix.

RIN-PREP-002Check Sudo Access

Signifies that the sudo password is incorrect.

Enter the sudo password, the non-root password of

the securonix account.




TypeTroubleshooting

RIN-PREP-003Connection Error

Signifies that there is a connection error between

RIN and SNYPR.

1. Open installer.properties using the following command:

vi installer.properties

2. Press i on your keyboard to display the Terminal application in the Insert mode. The Terminal application displays "Insert" when the mode is enabled.

3. Enter or verify the values for the settings in installer.properties.


5. Type :wq! to save the changes.


sh validation.sh prepare-to-

instal



RIN Post Installation Issues


TypeTroubleshooting

RIN-POST-001

INGESTER_HOME set properly Ingester is Running

As the non-root user (securonix user) execute

the following commands:

l source ~/.bash_profile

l Start - sudo systemctl start scnx-ingester

l Status - sudo systemctl status scnx-ingester

l Stop - sudo systemctl stop scnx-ingester

l Restart - sudo systemctl restart scnx-

ingester

RIN-POST-002 Syslog Running



l Start - sudo systemctl start scnx-

syslog-ng

l Status - sudo systemctl status scnx-syslog-ng

l Stop - sudo systemctl stop scnx-syslog-ng

l Restart - sudo systemctl restart scnx-syslog-ng




TypeTroubleshooting

RIN-POST-003

Gateway running



l Start - sudo systemctl start scnx-gateway

l Status - sudo systemctl status scnx-gateway

l Stop - sudo systemctl stop scnx-gateway

l Restart - sudo systemctl restart scnx-gateway




TypeTroubleshooting

RIN-POST-004

Fetch Kafka Broker Details

Option1 : Perform the following steps:

1. Make sure Snypr application is running.

2. If firewall is running, ensure that the firewall has port 443 open or https service

enabled.

firewall-cmd --list-all

If port is not open, then run the following

command:

firewall-cmd --permanent --add-

port=443/tcp

Option2: Perform the following steps if you

want to turn off the firewall:

l systemctl stop firewalld

l systemctl disable firewalld

RIN-POST-005 SNYPR Console Access

Signifies the SNYPR application is not running.

Ensure that the SNYPR application is running.




TypeTroubleshooting

RIN-POST-006

Check Kafka Broker Network Access

Option1 : Perform the following steps:

1. Make sure Snypr application is running.

2. If firewall is running, ensure that the firewall has port 9093 open or https

service enabled.

firewall-cmd --list-all

If port is not open, then run the following

command:

firewall-cmd --permanent --add-

port=9093/tcp

Option2: Perform the following steps if you

want to turn off the firewall:

l systemctl stop firewalld

l systemctl disable firewalld




TypeTroubleshooting

NAAuthentication Checks

See Appendix A for the instructions to create

the ingestercloud.properties file.

l Token validation fails

l URL or token is not provided in the ingestercloud.properties file

NAKafka Publishing fails with SSL error

l If the Kafka Brokers are protected with SSL and they are using self signed certificates,

ensure the following:

a. The truststore and SSL config file, sslconfig.properties, located in the INGESTER_HOME/conf folder must be configured to point to the

truststore.jks

b. The public keys of the Kakfa brokers or the signing certificate must be imported to the truststore.jks. See Appendix A for instructions.

l If the Kafka Brokers are configured with mutual SSL authentication, a client

certificate must be imported into the

keystore for the Ingester. The SSL config

file, sslconfig.properties, located in the

INGESTER_HOME/conf folder, must be

configured to point to the ingester-

client.jks. See Appendix A for instructions.




TypeTroubleshooting

NAGateway is Down

You can check if the Gateway is running or

down from the SNYPR application. If the

Gateway is down, it is displayed in red color.

You can perform the following steps to

troubleshoot:

1. Check the gateway log by running the following command from Terminal:

tail -1234f //Gateway/logs/Gateway.lo

g

2. Review the logs to understand why the gateway is down.

3. Open the SSLConfig.properties file and verify if the Ingester path is correct. If it is incorrect, update the path.

4. Validate Ingester_Home by using the following commands:

echo $INGESTER_HOME

When you run the echo command, the

command prompt displays the installation path

of the Ingester.



RIN and Syslog Servers Issues

The RIN installer automatically starts the RIN and Syslog server once the installation is complete. This section explains how to start and stop services if there are any issues.

Follow these steps to start the RIN and the Syslog Service:

Note: When you are manually restarting the RIN, you must first restart the

Gateway using the following command: sudo systemctl start scnx-

gateway .

1. Start RIN as a securonix user with the following command:

sudo systemctl start scnx-ingester

2. Check the RIN status to confirm that it has started with the following command:

systemctl status scnx-ingester

If there is an error or you want to check RIN logs, use this command:

tail -1234f /$INGESTER_HOME/logs/Ingester.log

3. Start the Syslog server as a securonix user with the following command:



lsudo systemctl start scnx-syslog-ng

To stop or check the status of the Syslog server, use the following command:

sudo systemctl stop scnx-syslog-ng

systemctl status scnx-syslog-ng

To check the logs for the Syslog server, use the following command:

journalctl -f -u scnx-syslog-ng -n 1234

RIN Log File Issues

To troubleshoot or examine the RIN log file, use this command:

tail -1234f /logs/Ingester.log

Generally, the default log level is set to debug in the RIN log file. If you would like to define a custom log level, change the log4j2.xml log level to trace. The file is available

at INGESTER_HOME/conf/log4j2.xml.


Appendix A: Configure Proxy Setup

Appendix A: Configure Proxy Setup If the RIN is behind a web proxy server, you have to configure the HTTP or HTTPs

parameters for the following RIN files:

l /etc/systemd/system/scnx-ingester.service

l /etc/systemd/system/scnx-gateway.service

l /Securonix/Ingester/bin/runingester.sh

l /Securonix/Gateway/rungateway.sh

Add or update the above files with the following information:

Parameters Type Description

HTTP

Add or update the following settings:

l Dhttp.proxyHost=

l Dhttp.proxyPort=

HTTPS

Add or update the following settings:

l Dhttps.proxyHost=

l Dhttps.proxyPort=

Sample HTTP and HTTPS SettingsBelow are the HTTP and HTTPS configurations:

Files HTTP Configuration Sample

Ingester Service file

ExecStart=/bin/sh -c '${JAVA_HOME} -

Dhttp.proxyHost= -Dhttp.proxyPort=

-cp ${INGESTER_HOME}/lib/ingester-6.2.jar:${INGESTER_

HOME}/lib/* com.securonix.ingester.Main -mode:cloud'



Files HTTP Configuration Sample

Gateway Service file



-XX:+UseG1GC -XX:+UseStringDeduplication -cp ${GATEWAY_

HOME}/snypr-gateway-1.0.jar:${GATEWAY_HOME}/lib/*

com.securonix.snypr.gateway.SnyprGateway ${GATEWAY_

HOME}/conf'

runingester.sh file

/Securonix/Ingester/Java/jre/bin/java -


-cp /Securonix/Ingester/lib/ingester-

6.2.jar:/Securonix/Ingester/lib/* com.securonix.ingester.Main -

mode:cloud

rungateway.sh file

/Securonix/Ingester/Java/jre/bin/java -Dhttp.proxyHost= -Dhttp.proxyPort= -XX:+UseG1GC -XX:+UseStringDeduplication -cp snypr-gateway-1.0.jar:lib/* com.securonix.snypr.gateway.SnyprGateway conf

Files HTTPS Configuration Sample

Ingester Service file


Dhttps.proxyHost= -

Dhttps.proxyPort= -cp ${INGESTER_

HOME}/lib/ingester-6.2.jar:${INGESTER_HOME}/lib/*

com.securonix.ingester.Main -mode:cloud'

Gateway Service file


Dhttps.proxyHost= -

Dhttps.proxyPort= -XX:+UseG1GC -

XX:+UseStringDeduplication -cp ${GATEWAY_HOME}/snypr-

gateway-1.0.jar:${GATEWAY_HOME}/lib/*

com.securonix.snypr.gateway.SnyprGateway ${GATEWAY_

HOME}/conf'



Files HTTPS Configuration Sample

runingester.sh file

/Securonix/Ingester/Java/jre/bin/java -

Dhttps.proxyHost= -

Dhttps.proxyPort= -cp

/Securonix/Ingester/lib/ingester-6.2.jar:/Securonix/Ingester/lib/*

com.securonix.ingester.Main -mode:cloud

rungateway.sh file

/Securonix/Ingester/Java/jre/bin/java -Dhttps.proxyHost= -Dhttps.proxyPort= -XX:+UseG1GC -XX:+UseStringDeduplication -cp snypr-gateway-1.0.jar:lib/* com.securonix.snypr.gateway.SnyprGateway conf


IntroductionServer RecommendationPrerequisites

Remote Ingester Node InstallationUpgrade to RIN 6.3.1Remote Management for RINCommunication FlowManage RIN in SNYPRConfigure Syslog Filters() and Source() in Activity Import

Network Performance TuningReference Server ConfigurationRIN Tuning ProcessBest Practices for Network TuningTroubleshooting Common Errors

Uninstall the RINTroubleshoot the RINAppendix A: Configure Proxy SetupSample HTTP and HTTPS Settings

BookmarksCreatesudoCentralized_ManagementActivity_Import

cloud rin installation guide · 12/7/2020 · the rin can be installed using the command line or...

Documents