logger reporting, the whole reporting and nothing but … – describes which events to include in...

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Logger reporting, the whole reporting and nothing but the reporting Johnny Khoury, CISSP Global Field Engineer Aaron Kramer, CISSP, CEH, Global Field Engineer, @SecGeek #HPProtect

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 2

Agenda • Define report, query, parameter (Aaron) • Managing report jobs (Johnny) • Compare tuned and non-tuned reports (Johnny) • Good vs. bad SQL queries (Johnny) • Sharing reports: Export, import (Aaron) • Running report using Logger API (Aaron)


Define report, query, parameter



Reporting provides repeatable, schedulable, summarization and detail of events • Logger report

– Consists of a query and a time range over which to run that query – Can be run on demand via UI, on a schedule, or over the Logger API – Output formats include HTML, PDF, MS Excel, CSV, MS Word, Interactive HTML, XML – Use Logger “Report Explorer” to view, copy, modify, run



Query – describes which events to include in the report, and how (if) to summarize • Logger query

– Can be created using GUI or writing SQL directly – SQL = Structured Query Language – More than 130 ship with Logger, can copy, modify – Refer to MySQL 5.1 syntax as a guide – Use Logger “Query Explorer” to view, copy, modify, test



Query – describes which events to include in the report, and how (if) to summarize



Parameter – optional part of a query, a ‘substitute-able’ field • Logger parameter

– Want to include the sourceUserName in a report, but the value you want could be anyone, not known in advance

– Create a UserName parameter, define length, type of data – Use the parameter in the query, which is used in the report


Managing report jobs


Managing report jobs

• Avoid scheduling too many reports to run at the same time • Schedule reports during non peak hours • Leverage dynamic time frame for running report. No need to schedule all reports to run

“00:00:01 am” for the previous day • For example schedule report to run at 2 am for the previous day • Start ( $NOW -26h) End ($Now -2h)


Report Jobs Execution Status: (job summary)


Tuned vs. non-tuned reports Good vs bad SQL queries


Logger report and general performance

The performance of a report SQL query will be affected by a combination of the following factors:

• Data volume – the larger the data volume, the more data needs to be processed • Data distribution – this includes uniqueness • Server load : This includes the number of searches and reports running concurrently • Query complexity – queries can be crafted to be more optimal. This will differ on a case-by-case basis


Brief background about Logger reports and UI search

Similarities: Both Logger UI Search and Logger MySQL run search in the backend to extract the data Both rely on Logger MySQL plugin to extract data from the storage Engine Both Logger Report and UI Search condition are translated into SQL-like condition (with the exception of

regex based search)

Select events.arc_name , events.arc_sourceAddress from events where name like ‘Failed%’ group by events.arc_name , events.arc_sourceAddress; vs

name contains "Failed*" | chart count by name sourceAddress Key difference: Logger report use sql query functions ( GROUP, ORDER, LIMIT ) aggregate and sort Logger data Logger UI Search use search command (TOP, CHART, SORT, DEDUP) to sort and aggregate the results


MySQL workflow

MySQL -storage engine workflow

* The JDBC connection is not open. Each search is giving search session ID * The results of search/sql output is stored in Temp table

Select …

from arc_event Where …

Order by …

Logger server Logger

MySQL plugin

Request for Metadata

Metadata

rows

JDBC for SQL request

(session id)

MySQL

Storage engine


1) Use Indexable fields in the SQL Query

Why are my reports are slow ? If any of the fields selected in the sql query are not indexed, the report speed will default to the slow non-index search. (By default 60 fields are indexed, they are listed under the search option) Bad query select events.arc_deviceHostName, events.arc_requestUrlQuery, events.arc_requestMethod, events.arc_userName, events.arc_CategoryOutcome, from events where events.arc_sourceAddress="10.10.10.12";

( Note events.arc_requestUrlQuery and events.arc_requestMethod are not indexed by default)


2) Leverage the super indexed fields in the WHERE CONDITION for the SQL

SuperIndex the report – With the introduction of Super-Index fields (Bloom Filters) in Logger 5.5 , there are 15 default fields that

users can select to improve the search speed significantly – The superindex fields need to be entered in the WHERE condition

deviceEventClassId deviceProduct deviceVendor destinationHostName

destinationPort destinationAddress destinationUserId destinationUserName

deviceAddress deviceHostName sourceHostName sourcePort

sourceAddress sourceUserId sourceUserName


Continue – leverage super-index fields

Example: no Super Index fields in the WHERE Condition select events.arc_sourceAddress, events.arc_sourceUserName, events.arc_requestMethod, SUM(events.arc_baseEventCount) , HOUR(events.arc_endTime) as "Hour“ FROM events WHERE events.arc_requestMethod = "GET“ AND events.arc_applicationProtocol CONTAINS "http“ GROUP BY events.arc_sourceAddress, events.arc_sourceUserName, events.arc_requestMethod, Hour ORDER BY HOUR

Good Query: Faster query using Super-Index fields in the WHERE condition select events.arc_sourceAddress, events.arc_sourceUserName, events.arc_requestMethod, SUM(events.arc_baseEventCount) ,HOUR(events.arc_endTime) as "Hour” FROM events WHERE events.arc_deviceVendor="Apache“ AND events.arc_deviceHostName = "buildserver“ GROUP BY events.arc_sourceAddress, events.arc_sourceUserName, events.arc_requestMethod, Hour ORDER BY HOUR


3) Use Limit option where possible

Another time and resource saver that can improve the SELECT statement to run faster and consume less memory cache is the LIMIT option. Example - Top 10 source Address select events.arc_sourceAddress, events.arc_sourceUserName, SUM(events.arc_baseEventCount) as "COUNT“ FROM events WHERE events.arc_deviceVendor="Apache" AND events.arc_sourceAddress IS NOT NULL GROUP BY events.arc_sourceAddress, events.arc_sourceUserName ORDER BY COUNT desc limit 10;

( BY default order will sort is asc order)


4) Avoid using sub-query if possible (slow)

– Sub queries are not handled well in MySQL by default. – The Sub-query can consume more space in memory and disk.( 2 temp tables to scan) Bad query select arc_eventTime, arc_name, arc_sourceAddress, arc_sourceHostName, arc_sourceProcessName, arc_sourceUserName, arc_destinationUserId, arc_destinationUserName from events where events.arc_sourceAddress in ( select distinct(events.arc_sourceAddress) from events where events.arc_sourceAddress != '192.168.36.34' AND events.arc_sourceAddress != ‘192.168.35.23’ AND events.arc_sourceAddress != '192.168.37.34’)

Good query select arc_eventTime, arc_name, arc_sourceAddress, arc_sourceHostName, arc_sourceProcessName, arc_sourceUserName, arc_destinationUserId, arc_destinationUserName from events where events.arc_sourceAddress != '192.168.36.34' AND events.arc_sourceAddress != 192.168.35.23 AND events.arc_sourceAddress != '192.168.37.34


5) GROUP BY with order by & GROUP BY... HAVING

GROUP BY is an aggregate function that operate on set values. It is equivalent to group on all rows based on selected fields. There are couple options to utilize GROUP BY : a) GROUP BY optimization with ORDER BY. select events.arc_sourceAddress, events.arc_destinationUserId, events.arc_categoryOutcome, SUM(events.arc_baseEventCount) as "COUNT", HOUR(events.arc_endTime) as "HOUR" FROM events WHERE events.arc_deviceVendor="Apache" AND events.arc_deviceHostName = "buildserver.dev" GROUP BY events.arc_sourceAddress, events.arc_destinationUserId, events.arc_categoryOutcome ORDER BY HOUR


Continue GROUP BY (1 page vs. 15 page)


Continue: Group BY …. HAVING

b) GROUP BY ..... HAVING Adding filter condition to aggregation for summary results. Report Failed logins attempt more than 3 time select events.arc_destinationUserName, COUNT(arc_baseEventCount) as "COUNT“ from events where events.arc_categoryBehavior="/Authentication/Verify" and events.arc_categoryOutcome="/Failure" Group by events.arc_destinationUserName Having COUNT > 3;


5) Use few columns where possible. Size matters

In the MySQL world, size is everything Avoid including too many fields in the SELECT statement. Use types that are smaller in size if possible. DestinationZoneUri=/All Zones/ArcSight System/Private Address Space Zones/RFC1918: 10.0.0.0-10.255.255.255 . The size of this value =89 bytes. msg =Network: A user or computer logged on to this computer from the network. (73 bytes) name=An account failed to log on. ( 29 bytes)


CONTINUE

Too many fields in this query: SELECT events.arc_deviceVendor, events.arc_deviceProduct ,events.arc_destinationZoneURI ,events.arc_name,

events.arc_destinationAddress , events.arc_destinationHostName, events.arc_destinationUserName, events.arc_sourceUserName ,events.arc_categoryOutcome , events.arc_endTime FROM EVENTS WHERE (events.arc_deviceVendor = "Microsoft" AND events.arc_deviceEventClassId = "Security:540")

order by events.arc_destinationUserName

For faster query: We can remove events.arc_deviceVendor,events.arc_destinationZoneURI , events.arc_deviceProduct ). SELECT ,events.arc_name, events.arc_destinationAddress , events.arc_destinationHostName, events.arc_destinationUserName, events.arc_sourceUserName , events.arc_categoryOutcome , events.arc_endTime FROM EVENTS WHERE (events.arc_deviceVendor = "Microsoft" AND events.arc_deviceEventClassId = "Security:540")

order by events.arc_destinationUserName


8) Use NOT NULL if you can

NULL columns require additional space in the row to record whether their values are NULL. Some of the device hostnames do not populate with a hostname, so it can throw the device counts off.

Select …. WHERE events.arc_deviceHostName IS NOT NULL GROUP BY events.arc_deviceHostname


Sharing Logger reports



Sharing reports increases in value exponentially* (not a financial commitment) • Logger lets you bundle up parameters, queries, and reports, and export them to a file • Logger lets you import report bundle(s) from others • These bundles are called CAB files (file extension is .CAB), along with a resource file called a .CONF file • For the security-minded, no digital signing exists, no risk assessment is implied • Other option would be to post the actual query used, and a screen shot of the report



Logger export • Encouraged to also post the actual query used, and a screen shot of the report • Reporting, Administration, iPackager

– “import” items from the Logger Report Server – build the .CAB file



Logger import • See if someone else has written the report you want, or something like it • Look at their screen shot, or query • Copy n paste OR obtain their .CAB file

– Many times copy and paste changes important characters such as quotes ‘ “ ` • Reporting, Administration, Deploy Report Bundle

– Upload .CAB file to Logger Server • Typical .CAB file is small: from 10KBytes to 100KBytes to 200 KBytes


Run reports with Logger API


Run reports with Logger API

Logger API is a programmatic way to run Logger reports (and Logger search, too!) • Logger API is fully documented in “Logger_WebServicesAPI_5.5.pdf”


Resources

Additional resources • Logger 5.5 Administrators Guide (PDF) https://protect724.hp.com/community/arcsight/productdocs/logger • My SQL 5.5 Reference: http://dev.mysql.com/doc/refman/5.5/en/functions.html • Logger API PERL code: https://protect724.hp.com/message/24390#24390

https://protect724.hp.com/community/arcsight/productdocs/logger

http://dev.mysql.com/doc/refman/5.5/en/functions.html

https://protect724.hp.com/message/24390


For more information

Attend these sessions

• TT3132, Army of loggers: : Searching for that needle at billions of EPS

• TT3154, : Renovation of the HP ArcSight Logger user interface

• TT3589, Enriching HP ArcSight Logger search with third-party information

Visit these demos

DEMO 3526 – HP ArcSight Logger

After the event

Contact your sales rep

Visit http://protect724.hp.com

Your feedback is important to us. Please take a few minutes to complete the session survey.

http://protect724.hp.com/





Please fill out a survey. Hand it to the door monitor on your way out.

Thank you for providing your feedback, which helps us enhance content for future events.

Session TB3123 Speaker Johnny Khoury, Aaron Kramer

Please give me your feedback


Thank you

logger reporting, the whole reporting and nothing but … – describes which events to include in...

Documents