arcsight connector health check
TRANSCRIPT
-
8/10/2019 ArcSight Connector Health Check
1/18
-
8/10/2019 ArcSight Connector Health Check
2/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2
Agenda
HP ArcSight Connector health check
What is a health check?
Health check steps by ArcSightcomponent
Connectors
Connector Appliances
Q & A
-
8/10/2019 ArcSight Connector Health Check
3/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Health Check overview
-
8/10/2019 ArcSight Connector Health Check
4/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4
What is a health check?
Purpose
The purpose of performing a health check is to identify and remove performbottlenecks to enable top performance of the HP ArcSight implementation
issues can result in major performance degradations over time impacting s
availability and user satisfaction. Performing regular health checks will ide
issues allowing them to be remediated quickly and ensure continued top p
of the HP ArcSight implementation.
In a nutshell
A Health Check consists of common administrative tasks to verify the ArcS
solution is configured and performing optimally.
-
8/10/2019 ArcSight Connector Health Check
5/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Health Check stepsby ArcSight component
Note: Its impossible to cover every scenario in this presentation,so only the common checks will be discussed.
-
8/10/2019 ArcSight Connector Health Check
6/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6
Health check steps by ArcSight component
1Lo
CPU
Che
Sea
Cus
Che
Rec
Sto
Inde
Con
Sch
Eve
Bac
Log
Eve
Net
Onl
(On
Log
ESM Databaseand storage
DBCheck and Oracle RDA
Database Performance
Statistics Dashboard Check
Partition Check (Oracle)
Trend Jobs Check
Hardware and Operating
System Check CPU and Memory Utilization
Check
Oracle version and patch level
check
Oracle alert log check
Oracle memory parameters
check
ESM Database Storage Check
ESM Manager
Event Throughput Dashboard Check
Current Event Sources Dashboard Check
Hardware and Operating System Check
CPU and Memory Utilization Check
ESM Manager JVM (memory) Utilization
Check
Data Monitor Utilization Check
Active List/Session List Utilization Check
Rules Engine Check
Event Persistence (insertion) Performance
Check
Error Check
Scheduled Task Check
server.properties Check
Agent and Console Threads Check
Connectorappliances
Version Check
CPU and Memory Check
Network Settings
Check
Configuration Backup
Check
Connectors
Up/Down Check
(Connector or Container)
Version Check
Connector Event RateCheck (by EPS)
Cache Check
Logs Check
Configuration Check
Connectors
Tip: Check each ArcSight Component by the order of the Event Flow
Its just simple plumbing!!!
-
8/10/2019 ArcSight Connector Health Check
7/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
Connectors
Connector (or Container) Up/Down Check
Connector Version Check
Are there any Connectors running a version older than ~1 year?
A minimum version of 4.8.1 is required to leverage the ESM v5.2 schema.
-
8/10/2019 ArcSight Connector Health Check
8/18
-
8/10/2019 ArcSight Connector Health Check
9/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9
Connectors (cont.)
Connector Logs Check ../current/logs/agent.out.wrapper.log
Java Heap Memory Utilization
Memory utilization
Frequency of Full GCs
Memory in Red Zone alerts
Unexpected Connector restarts
Connectivity errors
End Devices
ArcSight Destinations
../current/logs/agent.log
Parsing errors
DOSProtector
Chronic WARN and ERROR messages
-
8/10/2019 ArcSight Connector Health Check
10/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10
Connectors
Connector Logs Check (cont.) Use Connector LogFu to graph the event
flow and memory utilization ../current/bin/arcsight agent logfu a
-
8/10/2019 ArcSight Connector Health Check
11/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11
Connectors (cont.)
Connector Configuration
Check Destination Settings
Are there more than 2 Destinations on each
Connector?
Too many Destinations can negatively
impact performance of a Connector.
Common problems found:
Networks and CustomerURI are not applied
on every Connector
Fields-based Aggregation is not properlyapplied (by Connector Type)
No tuning (Filter Out) applied on high EPS
Connectors
Settings are not the same on every
Destination (ESM, Logger, etc.)
-
8/10/2019 ArcSight Connector Health Check
12/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12
Connectors (cont.)
Connector Configuration Check (cont.)
Only check the following on problematic Connectors discovered in previous checks
../current/user/agent/agent.properties
Optimal settings are different for each Connector type
High EPS Connectors (>1200 EPS) such as Syslog, WUC, CheckPoint, and Blue Coat can be tweaked quite a bit here
../current/user/agent/agent.wrapper.conf
Only increase the Java Heap size if memory issues were found in agent.out.wrapper.log
Default Java Heap is 256MB
Maximum configurable Java Heap is 1024MB (1 GB)
Reminder: If you have 50+ Connectors in your environment, try to stay focused on problematic Connectors!
-
8/10/2019 ArcSight Connector Health Check
13/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13
Health check steps by ArcSight component
1Lo
CPU
Che
Sea
Cus
Che
Rec
Sto
Inde
Con
Sch
Eve
Bac
Log
Eve
Net
Onl
(On
Log
ESM Databaseand storage
DBCheck and Oracle RDA
Database Performance
Statistics Dashboard Check
Partition Check (Oracle)
Trend Jobs Check
Hardware and Operating
System Check
CPU and Memory Utilization
Check
Oracle version and patch level
check
Oracle alert log check
Oracle memory parameters
check
ESM Database Storage Check
ESM Manager
Event Throughput Dashboard Check
Current Event Sources Dashboard Check
Hardware and Operating System Check
CPU and Memory Utilization Check
ESM Manager JVM (memory) Utilization
Check
Data Monitor Utilization Check
Active List/Session List Utilization Check
Rules Engine Check
Event Persistence (insertion) Performance
Check
Error Check
Scheduled Task Check
server.properties Check
Agent and Console Threads Check
Connectorappliances
Version Check
CPU and Memory Check
Network Settings
Check
Configuration Backup
Check
Connectors
Up/Down Check
(Connector or Container)
Version Check
Connector Event RateCheck (by EPS)
Cache Check
Logs Check
Configuration Check
Connector AppliancesTip: Check each ArcSight Component by the order of the Event Flow
-
8/10/2019 ArcSight Connector Health Check
14/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14
Connector appliances
Connector appliance version check
Is the version outdated?
Are there any known issues with the current version?
Connector appliance CPU and memory
check
Review the following for excessive utilization:
CPU utilization is continuously above 70-80% in Logger Dashboard
EPS In is continuously above 5,000 EPS (a single C5400 is designed
for 5,000 max EPS)
Check the Connector Appliances Monitor Dashboards for unusual
peaks or drops
Check the System Process Status section of the Connector Appliance
If possible, SSH to the Connector Appliance and run commands such
as top, df, ifconfig, etc. to perform a deeper dive at the OS level
Connector appliance network
check
Common problems to check:
Incorrect duplex settings on the network interfa
DNS or NTP not configured properly
Connector appliance configur
check
The daily Configuration Backup job should
all Connector Appliances.
-
8/10/2019 ArcSight Connector Health Check
15/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Additional resources
-
8/10/2019 ArcSight Connector Health Check
16/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16
My favorite resources for keeping ArcSight he
1. Any HP Protect presentation on ArcSight best practices or troubleshooti
https://protect724.arcsight.com2. KB Articles on the HP Support Site
3. Solutions listed in previous Support Tickets
4. HP ArcSight University
5. HP ArcSight product documentation
https://protect724.arcsight.com/https://protect724.arcsight.com/ -
8/10/2019 ArcSight Connector Health Check
17/18
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you
-
8/10/2019 ArcSight Connector Health Check
18/18