syslog connector performance tuning · arcsight cef encrypted syslog (udp) ... supported only on...

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Syslog Connector Performance Tuning Girish Mantry, Moehadi Liang Technical Solutions Consultants

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 2

Syslog Connector Performance Tuning Agenda In this session we will take a look at • Syslog connector variants • Connector components and operation • Stages in the event flow • Performance bottlenecks and tuning at each stage • Out of memory problems and tuning • Customer cases • General recommendations


Syslog Connector Performance Tuning

Syslog connector variants, components, operation and event flow


Syslog Connector Performance Tuning Syslog Connector Variants

Network Listeners

File Readers

ArcSight CEF Encrypted

Syslog (UDP)

Syslog NG

Daemon

Syslog Daemon

Syslog Pipe

Syslog File

UDP Raw TCP

Default port 514

UDP Raw TCP

TLS Default port 1999

UDP Symmetric Key

Encryption Default port 514 Only CEF format

Unix Pipe

Regular File

Supported on all platforms Configurable interfaces and ports

Supported only on unix platforms Work in conjunction with the native syslog daemon


Syslog Connector Performance Tuning Syslog Connector Components

Device Type 1

Device Type 2

Device Type N

Subagent ESM

Logger

Queue Cache

Subagent

Subagent

Raw Events

ESM Transport

Logger Transport

Cache

C1 C2

Main Flow

C1 C2

Destination Flow

C1 C2

Destination Flow Parsed Events

Processed Events

Note: Queuing only applies to network listeners and not for file readers


Syslog Connector Performance Tuning Event Flow

Event Reception Event Queuing Event Parsing Event Processing Event Transport

• Receives network packets on UDP/TCP sockets

• Extracts human readable syslog raw events from network packets

• Raw events are written to a queue of files on the file system in the order in which they are received

• Raw events are picked up from the file queue in a FIFO manner and parsed using regular expressions

• Information from device log formats normalized into Arcsight event format

• Normalized events are categorized and processed in many ways useful for correlation and asset modeling

• Events are batched, filtered or aggregated as required for efficiency

• Enriched Arcsight events are sent to ESM/Logger destination

• Events cached when destinations are down and resent when they are back up



Performance Bottlenecks in the Event Flow and Tuning


Syslog Connector Performance Tuning Event Reception • Choice of Transport Protocol

– UDP performs better on reliable networks – Use Raw TCP on unreliable networks – Use TLS for encrypted transport with Syslog NG

• Bottleneck (when dealing with Raw TCP or TLS) – Java applications do not know when a client closes the connection with a FIN – Connections remain idle in a CLOSE_WAIT state until closed explicitly by the application

– Idle connections can grow over a period of time and can exceed the connector limit or OS limit – Happens faster with large number of devices or with devices that create new connections frequently

• Tuning

Parameter Default Recommendation

tcppeerclosedchecktimeout -1 Set it to 30000 msec or higher to tell the connector to check for connections closed by peer proactively and close them on the connector side as well

tcpmaxsockets 1000 Increase it higher as required to accommodate simultaneous connections from a large number of devices


Syslog Connector Performance Tuning Event Queuing Raw events received over the network are written to a file queue consisting of a certain number of files of fixed size • Bottleneck

– With high event volumes, file queue can build up faster leading to significant delays – When file queue becomes full, connector starts dropping events

• Tuning – Enable syslog parser multithreading (may need to follow up with memory increase if required) – Increase the file queue size


syslog.parser.multithreading.enabled false Set it to ‘true’ to enable multithreading

syslog.parser.threadcount -1 Set it to a specific number on a single processor machine. You can do the same on a multiprocessor machine or leave it for connector to decide based on the number of processors

syslog.parser.threadsperprocessor 1 Takes effect only when the threadcount is set to -1. Leave it at 1 or increase it as required. Total number of threads = number of processors * threadsperprocessor

filequeuemaxfilecount 100 Increase this parameter to increase the number of files in the file queue

filequeuemaxfilesize 100000 Specified in bytes. Increase this parameter to increase the size of each file in the file queue


Syslog Connector Performance Tuning Event Parsing • Inspection and Device Type Detection

– Multiple subagents with one subagent per device type with a parser that has a regex to match something unique in the log – Subagent parsers are ordered such that specific regexes come ahead of generic ones to detect device types accurately – Connector inspects messages from senders applying regexes in the order to detect the device type and associates the subagent with the sender

when a match is found. A single sender could be associated with multiple device types and subagents

– Associated subagent parsers are used to parse messages from a sender and inspection process is not reapplied unless a message from a new device type is encountered from the same sender

– Syslog senders and their associated subagent types can be seen in current/user/agent/syslog.properties

• Bottleneck – Inspection process involving regex matching could be expensive because connector has more than 100 subagents

• Tuning – If you are sure of device types in your environment, you can restrict the subagent list by following properties


usecustomsubagentlist false Set it to true to make the connector consider the customized subagent list

customsubagentlist List of subagents (>100) Set it to the restricted subagent list based on device types in your environment. Preserve the original relative order of subagents not to affect the accuracy of subagent detection


Syslog Connector Performance Tuning Event Parsing - continued • Regular expressions in parsers

– Bottleneck A badly written regular expression in the parser can be a big performance hit on the connector

– Optimization For supported device types, development went through optimizing the regular expressions in the respective parsers. If you are authoring your own

syslog flex connector parsers, consider the following guidelines. • Make your regexes generic only as much as needed. Specific regular expressions perform better than generic ones • Use generic greedy expressions like .* and .+ at the end and not in the beginning or middle of a regular expression. Replace them with non-greedy

equivalents like .*? and .+? with a clear character or token marking the boundary.

• Use of greedy expressions with more specific characters or meta characters is okay, ex:- \s+ for a continuous string of whitespace characters or \d+ for a continuous string of numerals or \w+ for a continuous string alpha numerals

• Maximum number of devices – Bottleneck

Connector allows up to a max of 5000 devices and does not process events from newer devices once this limit is reached

– Tuning


syslog.max.device.count 5000 Increase it as required to match the number of devices in your environment


Syslog Connector Performance Tuning Event Processing • Agent Batching

– Batch size controls how many events go together from component to component in the event flow and eventually to the destination – Doubling or tripling default size of 100 could help improve the performance internally as well as over networks with latency – Do not increase beyond that because it could have a negative impact by increasing memory requirements to hold the batches

• Categorization – Categorization files for different device types are loaded into memory and some of those can be big – Connector base memory usage can be high when dealing with a large number of device types

– Java heap space may need to be bumped up

• External Map File Processing – External map file query is executed for every batch of events

– Make sure the query is simple and returns fast, if you are using this feature

• Connector Filtering – Make sure that the filter condition is optimized and not extremely complex


Syslog Connector Performance Tuning Event Processing - continued • Field Based Aggregation

– Groups events with same values in specified fields into buckets and produces aggregated events on time interval expiry or reaching event threshold – Restrict the field set to minimum required and choose an optimal event threshold value to keep the number of event field comparisons low – Choose an optimal time interval not to block the event flow for too long

– Avoid using ‘preserve common fields’ setting in a high event volume environment

• Name Resolution – Name resolutions are done in background threads and the event flow is not normally blocked for the answers to come back

– If the ‘Wait For Name Resolution’ feature is enabled, then the event flow is blocked for a certain timeout period for the answers to come back – Do not enable ‘Wait For Name Resolution’ feature in an environment requiring frequent resolutions


Syslog Connector Performance Tuning Event Transport Event caching can occur for a number of reasons - network latency and problems in the destination are the common reasons • Bottleneck

– Excessive caching can cause delays in events reaching their destination – When cache becomes full, connector starts dropping events

• Tuning – Enable transport multithreading (except when the root cause is a problem in the destination) – For the logger smart message transport, turn on the https persistent connection feature

– Increase the cache size to hold events for longer in the cache and prevent loss of events


http.transport.threadcount 1 Applies only to the ESM transport. Increase it by small increments as required.

transport.loggersecure.threads 1 Applies only to the logger secure transport. Increase it by small increments as required.

transport.loggersecure.connection.persistent false Applies only to the logger secure transport. Change it to true for reusing the existing HTTPS connections and not tear them down for every batch of events

Cache Size 1GB Increase it as required up to a limit of 50GB. This is a destination setting which can be configured using ESM console, connector appliance GUI or local connector setup wizard.



Out of Memory Problems and Tuning


Syslog Connector Performance Tuning Java Process Memory and Management Memory allocated to a java process consists of Heap Space and Native Memory • Heap space is allocated as instructed by java run time parameters

– -Xms (Initial heap size), -Xmx (Maximum heap size), 256 MB by default on connectors

• Native memory size = Process Memory size – Size of Heap Space • Garbage Collection reclaims the memory of unused objects

– Minor collections (GC), reclaims memory in YOUNG generation and moves survivors into OLD – Major collections (Full GC), reclaims memory in all of the Heap space, takes much longer

– JVM stops the application threads during GC or Full GC – Frequent Full GCs affects application performance severely

• A clear indicator for the need to increase the maximum heap size

• Out of memory errors can happen in any of these memory areas • Memory limitations in 32 bit connector build

– Total addressable space is 4GB, Kernel space ranges from 1GB to 2GB depending on OS

– User space available for process is 2GB to 3GB depending on OS – Limits exist on max heap space: 1GB (connector appliances), 1.5 GB (Windows), 2 GB (Unix)

• Use 64 bit connector build for higher memory

Process Memory

YOUNG Generation Newly created objects

OLD Generation

Old objects surviving minor GCs

PERMANENT Generation

Classes, methods, etc

Heap Space

Native Memory

Code Generation Socket Buffers

Thread Stacks Direct Memory Space JNI Code

Garbage Collection JNI Allocated memory


Syslog Connector Performance Tuning Dealing with Java out-of-memory errors

Errors Root Cause and Recommendation

java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Requested array size exceeds VM limit

Garbage collection is unable to free up more space and memory could not be allocated for new objects • Increase the maximum heap size using -Xmx option in increments as required up to the limit • If this still does not help, there could be a potential memory leak or a bug – open a support incident supplying the logs and heap dumps

java.lang.OutOfMemoryError: PermGen space

Permanent generation area has become full due to loading many classes statically or creating dynamic classes or creating too many interned strings • Default max size of PermGen space is 64 MB. Increase it in small increments using -XX:MaxPermSize option

java.lang.OutOfMemoryError: Unable to create a new native thread

JVM is low on native memory and unable to create a new VM thread. Make more native memory available by • Reducing the heap space using –Xms and –Xmx options • Reducing the stack space of using –Xss option

Out of Memory Error (allocation.cpp:211), pid=16950, tid=1855142800

Displayed in the fatal error logs when the JVM crashes due to a malloc failure. The system could be out of physical RAM or swap space or the process size limit was hit on a 32 bit system. Take one or more of the following actions • Reduce memory load on the system or increase physical memory or swap space • Decrease the number of application threads, reduce the java heap space and stack space


Syslog Connector Performance Tuning Adjusting memory options On a software connector, add or edit settings in a file under current/user/agent folder • agent.wrapper.conf when running as a service • setmem.sh (Unix) or setmem.bat (Windows) when running as a standalone application. This file may have to be created if it does not already exist.

set ARCSIGHT_MEMORY_OPTIONS="-Xms256m -Xmx256m“ (Example only. Add or remove options as required inside the double quotes) export ARCSIGHT_MEMORY_OPTIONS (only on Unix)

On a connector appliance • Only heap space can be changed using a container command ‘Configure Memory Settings’

• Other settings can be changed using SSH or diagnostic tools file editor using the same mechanisms as for a software connector

Memory Type Running as service Running as a standalone application

Heap Space wrapper.java.initmemory=256 (initial heap size) wrapper.java.maxmemory=256 (maximum heap size)

-Xms256m –Xmx1024m” It is recommended to increase only the max heap size

Perm Gen Space

Add additional java parameters with adjusted indexes wrapper.java.additional.7=-XX:PermSize=64m wrapper.java.additional.8=-XX:MaxPermSize=128m

-XX:PermSize=64m -XX:MaxPermSize=128m It is recommended to increase only the max perm size

Stack space Add an additional java parameter with adjusted index wrapper.java.additional.9=-Xss=64k

-Xss=64k Default stack size is OS dependent. Adjust and observe. Too low a value can cause StackOverflowError



Customer Cases


Syslog Connector Performance Tuning Troubleshooting • Changing the transport protocol to UDP or Raw TCP did not help

• Could not reproduce the problem in house • Customer captured tcpdump packets and analyzed them using Wireshark

– Large number of “TCP Window Full” messages

– SEQ/ACK analysis showed that at times there is more than 10KB data in flight indicating that the receiver is too slow to process the incoming flood of packets

– TCP receive buffer and window sizes got reduced over time which contributed to the slow reception – Further enquiries revealed that the Syslog NG connector is receiving TLS data from 2 other sources

– With this new discovery of customer environment, problem could also be reproduced in house – Observed a high memory usage and Increased the heap space to1024 MB, but it did not help

Root Cause • Destination Syslog NG connector did not close TCP connections when sources closed connections • Growing TCP connections forces receive buffer size to be reduced causing slower reception

Solution • Set the ‘tcppeerclosedchecktimeout’ parameter to 30000 msec (half a minute) • This parameter tells the connector to proactively check and close any TCP sockets

Source Connector

Syslog NG Connecto

r

ESM

Logger

CEF Syslog

TLS

Customer Case 1- Problem CEF Syslog TLS destination was caching at only at 200eps, while ESM and Logger destinations did not cache for the same event rate

Syslog NG Source 1

Syslog NG Source 2

TLS



Syslog Connector

Logger

Customer Case 2- Problem Huge difference of event counts found between Fortigate Firewall and Logger via Syslog connector

Fortigate Firewall

queuing

caching

agent.out.wrapper.log:INFO | jvm 1 | 2012/12/05 11:35:29 | [Full GC agent.out.wrapper.log:INFO | jvm 1 | 2012/12/05 11:37:08 | [Full GC agent.out.wrapper.log:INFO | jvm 1 | 2012/12/05 11:38:52 | [Full GC agent.out.wrapper.log:INFO | jvm 1 | 2012/12/05 11:40:30 | [Full GC agent.out.wrapper.log:INFO | jvm 1 | 2012/12/05 11:42:06 | [Full GC agent.out.wrapper.log:INFO | jvm 1 | 2012/12/05 11:43:47 | [Full GC agent.out.wrapper.log:INFO | jvm 1 | 2012/12/05 11:45:32 | [Full GC agent.out.wrapper.log:INFO | jvm 1 | 2012/12/05 11:47:10 | [Full GC

Queue Rate(SLC) vs Events/Sec(SLC) Queue Drop Count Memory usage (Total vs Used)

Events/Sec(SLC) vs Throughput(SLC) Cache Size and Current Drop count Frequent Full GCs

Observations • Incoming event rate was much higher than the processing rate and connector was queuing heavily

• During peak hours, queuing has exceeded the size limit and dropped a huge number of events • Caching observed during peaks hours and some events were dropped when cache size limit is exceeded • High memory usage and frequent Full GCs were observed affecting the performance of the connector


Syslog Connector Performance Tuning Customer Case 2 – Solution Machines hosting the connectors were very powerful (64 bit Linux, 48 core CPU, 128 GB RAM, 600 GB hard disk) Actions Taken • Increased the java heap size to 2048 MB to reduce the frequency of full GCs • Enabled syslog parser multi-threading to keep up with the queuing rate

• Increased the file queue size from 100 to 2000 files of 10MB equivalent to 20 GB in total size to prevent dropping of events from file queue • Increased the cache size from 1GB to 10GB to prevent dropping of events from cache during peak hours • The above measured helped the performance of the connector significantly

– Where it did not help solve the problem completely, we asked the customer to split the event volume among multiple syslog connectors



Onboard Connector

ESM In USA

Customer Case 3- Problem Customer had loggers in UK and USA forwarding events to an ESM manager in US. Only the UK loggers were experiencing caching and event loss

caching

Observations • Time per Batch = roundtrip time taken for a batch of events to travel from logger to ESM and the

acknowledgment for the batch to come back from ESM logger

• The US logger took an average of 40 msec/batch and the UK logger took an average over 500 msec/batch • This large difference in the round trip time is indicative of network latency due to geographical distance and is

the root cause of caching in the UK logger

Logger in UK

Onboard Connector

Logger in USA

Solution • Enabled multithreading on the ESM transport with a thread count of 2, this showed an improvement in throughput

• Increased the thread count to 7 (number of processors in the CPU) and caching went away completely

USA Logger: Time per Batch ~ 40 msec UK Logger: Time per Batch > 500 msec



Recommendations


Syslog Connector Performance Tuning Some Recommendations • Evaluate the number of devices, device types and event volume early in your deployment cycle • Split the load among multiple connectors when the incoming event rate exceeds the achievable maximum

that varies based on the underlying platform and environment • When splitting the load, consider grouping the devices of same type to one connector and another type to a

different connector • Evaluate the total capacity of your machine and other processes running to determine the number of

connectors to install on a single machine • Cumulative heap size allocated to connectors and other java processes should be well below the total

memory available on the system • Use 64 bit syslog connector builds to overcome the out of memory errors


Please fill out a survey. Hand it to the door monitor on your way out.

Thank you for providing your feedback, which helps us enhance content for future events.

Session TB3248 Speakers Girish Mantry, Moehadi Liang

Please give me your feedback


Thank you

syslog connector performance tuning · arcsight cef encrypted syslog (udp) ... supported only on...

Documents