tibco loglogic universal collector user guide · tibco loglogic® universal collector (uc) is a...

TIBCO LogLogic® Universal CollectorUser GuideSoftware Release 2.5.0May 2014

Important Information | 2

Universal Collector

Important Information

SOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE. USE OF SUCHEMBEDDED OR BUNDLED TIBCO SOFTWARE IS SOLELY TO ENABLE THE FUNCTIONALITY (ORPROVIDE LIMITED ADD-ON FUNCTIONALITY) OF THE LICENSED TIBCO SOFTWARE. THE EMBEDDEDOR BUNDLED SOFTWARE IS NOT LICENSED TO BE USED OR ACCESSED BY ANY OTHER TIBCOSOFTWARE OR FOR ANY OTHER PURPOSE.

USE OF TIBCO SOFTWARE AND THIS DOCUMENT IS SUBJECT TO THE TERMS AND CONDITIONSOF A LICENSE AGREEMENT FOUND IN EITHER A SEPARATELY EXECUTED SOFTWARE LICENSEAGREEMENT, OR, IF THERE IS NO SUCH SEPARATE AGREEMENT, THE CLICKWRAP END USERLICENSE AGREEMENT WHICH IS DISPLAYED DURING DOWNLOAD OR INSTALLATION OF THESOFTWARE (AND WHICH IS DUPLICATED IN THE LICENSE FILE) OR IF THERE IS NO SUCH SOFTWARELICENSE AGREEMENT OR CLICKWRAP END USER LICENSE AGREEMENT, THE LICENSE(S) LOCATEDIN THE “LICENSE” FILE(S) OF THE SOFTWARE. USE OF THIS DOCUMENT IS SUBJECT TO THOSETERMS AND CONDITIONS, AND YOUR USE HEREOF SHALL CONSTITUTE ACCEPTANCE OF AND ANAGREEMENT TO BE BOUND BY THE SAME.

This document contains confidential information that is subject to U.S. and international copyright laws andtreaties. No part of this document may be reproduced in any form without the written authorization of TIBCOSoftware Inc.

TIBCO, Two-Second Advantage, and LogLogic are either registered trademarks or trademarks of TIBCOSoftware Inc. in the United States and/or other countries.

Enterprise Java Beans (EJB), Java 2 Platform Enterprise Edition (J2EE), Java Runtime Environment (JRE),Java Message Service (JMS), and all Java-based trademarks and logos are trademarks or registeredtrademarks of Oracle Corporation in the U.S. and other countries.

All other product and company names and marks mentioned in this document are the property of their respectiveowners and are mentioned for identification purposes only.

THIS SOFTWARE MAY BE AVAILABLE ON MULTIPLE OPERATING SYSTEMS. HOWEVER, NOT ALLOPERATING SYSTEM PLATFORMS FOR A SPECIFIC SOFTWARE VERSION ARE RELEASED AT THESAME TIME. SEE THE README FILE FOR THE AVAILABILITY OF THIS SOFTWARE VERSION ON ASPECIFIC OPERATING SYSTEM PLATFORM.

THIS DOCUMENT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS ORIMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY,FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT.

THIS DOCUMENT COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS.CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESE CHANGES WILLBE INCORPORATED IN NEW EDITIONS OF THIS DOCUMENT. TIBCO SOFTWARE INC. MAY MAKEIMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S) AND/OR THE PROGRAM(S) DESCRIBED INTHIS DOCUMENT AT ANY TIME.

THE CONTENTS OF THIS DOCUMENT MAY BE MODIFIED AND/OR QUALIFIED, DIRECTLY ORINDIRECTLY, BY OTHER DOCUMENTATION WHICH ACCOMPANIES THIS SOFTWARE, INCLUDING BUTNOT LIMITED TO ANY RELEASE NOTES AND "READ ME" FILES.

Copyright © 2002-2014 TIBCO Software Inc. ALL RIGHTS RESERVED.

TIBCO Software Inc. Confidential Information

TOC | 3

Universal Collector

Contents

Preface...............................................................................................................................7Related Documentation....................................................................................................................... 8Typographical Conventions................................................................................................................. 9Connecting with TIBCO Resources.................................................................................................. 10

Chapter 1 Introduction........................................................................... 11Overview............................................................................................................................................ 11

Chapter 2 Collecting Logs..................................................................... 13Real-Time File Logs.......................................................................................................................... 13

Collecting Single-line Messages.............................................................................................14Log File Rotation..........................................................................................................14

Collecting Multi-line Messages............................................................................................... 16Custom Multi-line Log Sources....................................................................................16

Windows Event Logs.........................................................................................................................17Local Collection.......................................................................................................................19Remote Collection...................................................................................................................20Filtering Windows Event Logs................................................................................................ 21

Syslog Logs....................................................................................................................................... 21Filtering Syslog Logs.............................................................................................................. 23

Remote Files......................................................................................................................................23UC Internal Logs............................................................................................................................... 23Creating and Configuring Log Sources.............................................................................................24

Add a New Log Source..........................................................................................................25Copy a Log Source................................................................................................................ 26Delete a Log Source.............................................................................................................. 27Creating Multiple Log Sources............................................................................................... 28

Create a CSV File....................................................................................................... 28Import Log Sources..................................................................................................... 29

Creating a Complete Configuration...................................................................................................29Edit Configuration General Settings....................................................................................... 30Add a New Configuration....................................................................................................... 31Open a Stored Configuration..................................................................................................32Activate the Configuration...................................................................................................... 33Save a Configuration.............................................................................................................. 34

Editing Log Sources.......................................................................................................................... 34Edit a Real-Time File Log Source..........................................................................................35Edit Multiple Real-Time Log Sources.....................................................................................38Edit a Windows Event Log Source........................................................................................ 39Edit Multiple Windows Event Log Sources............................................................................ 42Edit a Syslog Log Source.......................................................................................................43Edit Multiple Syslog Log Sources...........................................................................................46Edit a Remote File Log Source..............................................................................................47Edit Different Types of Log Sources...................................................................................... 50

Sorting Log Sources..........................................................................................................................51Create a New Tag.................................................................................................................. 52Apply a Tag............................................................................................................................ 53Remove a Tag........................................................................................................................ 54Sort Log Sources....................................................................................................................55

TOC | 4

Universal Collector

Chapter 3 Forwarding Logs................................................................... 57Creating a Syslog TCP or UDP Connection.....................................................................................57Creating an LMI Connection............................................................................................................. 59Creating a Connection in Authentication and/or Encryption Mode................................................... 60

Step 1: Get a Root Certificate Authority from your PKI......................................................... 62Step 2: Create a Certificate Signing Request........................................................................ 63

Using the Internal Tool................................................................................................ 63Using the OpenSSL..................................................................................................... 63

Step 3: Create a Valid UC Certificate using a CA and OpenSSL.......................................... 65Step 4: Import the Certificate into *.ks or *.p12..................................................................... 66Step 5: Configure the Forwarding Process............................................................................ 67

For *.ks.........................................................................................................................67For *.p12...................................................................................................................... 68For *.pem..................................................................................................................... 68Configure the Forwarding Process.............................................................................. 68

Step 6: Enable Secure Connection........................................................................................ 70Managing the list of Forwardings......................................................................................................71

Copying a Forwarding............................................................................................................ 72Deleting a Forwarding............................................................................................................ 73

Chapter 4 Monitoring UC Activities.......................................................75Starting UCMon Tool.........................................................................................................................75

To start UCMon from UC Console.........................................................................................75To start UCMon manually.......................................................................................................75

Summary Screen............................................................................................................................... 75Status Screen.................................................................................................................................... 77

Log Source Status.................................................................................................................. 78Forwarding Connection Status............................................................................................... 79

Metrics Screen...................................................................................................................................79Log Source Metrics.................................................................................................................81Forwarding Connection Metrics.............................................................................................. 82

Trends Screen................................................................................................................................... 82Log Source Trends................................................................................................................. 83Forwarding Connection Trends.............................................................................................. 84

RealTime Screen............................................................................................................................... 84Log Sources RealTime........................................................................................................... 85Forwarding Connection RealTime.......................................................................................... 86

Chapter 5 Command Line Interface...................................................... 87cert_mgt: Manage the Security Certificates...................................................................................... 87uc_checkConf: Check the Current Configuration..............................................................................88uc_createLogSources: Import and Create Several Log Sources at a time.......................................88uc_decodePwd: Decode Passwords for Windows Files................................................................... 89uc_encryptPwd: Encrypt Passwords for Windows Files................................................................... 89uc_monitor: UCMon Tool.................................................................................................................. 89uc_reload: Reload Configuration.......................................................................................................90uc_saveActiveConfAs: Save an Active Configuration.......................................................................90uc_switchTo: Make Configuration Active.......................................................................................... 91

Appendix A Sample Configuration Files...............................................93[UC Configuration] uc.xml................................................................................................................. 94

TOC | 5

Universal Collector

[LMI Connection]uldp-sampleCommented.uldp.xml.......................................................................... 95[LMI Connection] uldp-sampleCommentedAuthJks.uldp.xml............................................................ 97[LMI Connection] uldp-sampleCommentedAuthPem.uldp.xml.......................................................... 99[LMI Connection] uldp-sampleCommentedAuthPks12.uldp.xml......................................................101[Log Sources] file-sampleCommented.ls.xml.................................................................................. 103[Log Sources] syslog-sampleCommented.ls.xml.............................................................................106[Log Sources] wmi-sampleCommented.ls.xml.................................................................................108

Appendix B Regular Expressions....................................................... 111

TOC | 6

Universal Collector

Universal Collector

Preface

TIBCO LogLogic® Universal Collector (UC) is a software agent that collects logs from any device or logsource and forwards them to either an LMI Appliance using the proprietary Universal Log Data Protocol(ULDP) or to a Syslog server using the User Datagram Protocol (UDP), or Transmission Control Protocol(TCP). UC collects the information from four types of log sources: Syslog, Windows Event Logs, Real-TimeFile pull, or Remote File pull.

Preface | 8

Universal Collector

Related Documentation

The following documents contain information about Universal Collector:

• Release Notes — Provides specific release information including product information, new features andfunctionality, resolved issues, known issues, upgrade instructions, and any late-breaking information.

• Installation Guide — Provides instructions for installing, configuring, and uninstalling the UniversalCollector.

• User Guide — Describes how to collect and forward logs with the Universal Collector. In addition, theguide describes how to administer the Universal Collector.

• Online Help — Web Help is embedded in UC. Access Online Help by clicking the Help menu from the

main page. From the application, click on any screen to display the corresponding Help topic.

Preface | 9

Universal Collector

Typographical Conventions

Table 1: General Typographical Conventions

Convention Use

Bold font Indicates menu names, field names, button names, tab names and emphasis. Forexample: Click the Forwarding tab.

Code font Indicates file names, commands, and directory path. For example: <UC_HOME>\logs

italic font Italic font is used in the following ways:

• To indicate a document title. For example: See TIBCO LogLogic Universal CollectorUser Guide.

• To define new terms. For example: A keystore is a database of keys and certificates.• To indicate a variable in a command or code syntax that you must replace. For

example: MyCommand pathname.

Keycombinations

Key name separated by a plus sign indicate keys pressed simultaneously. For example:Ctrl+C.

Key names separated by a comma and space indicate keys pressed one after theother. For example: Esc, Ctrl+Q.

The note icon indicates information that is of special interest or importance, forexample, an additional action required only in certain circumstances.

The tip icon indicates an idea that could be useful, for example, a way to apply theinformation provided in the current section to achieve a specific result.

The warning icon indicates the potential for a damaging situation, for example, data lossor corruption if certain steps are taken or not taken.

Preface | 10

Universal Collector

Connecting with TIBCO Resources

How to Join TIBCOmmunity

TIBCOmmunity is an online destination for TIBCO customers, partners, and resident experts. It is a placeto share and access the collective experience of the TIBCO community. TIBCOmmunity offers forums,blogs, and access to a variety of resources. To register, go to http://www.tibcommunity.com.

How to Access TIBCO Documentation

After you join TIBCOmmunity, you can access the documentation here: http://docs.tibco.com.

How to Contact TIBCO Support

For comments or problems with this manual or the software it addresses, contact TIBCO Support asfollows:

• For an overview of TIBCO Support, and information about getting started with TIBCO Support, visit thissite:

http://www.tibco.com/services/support

• To contact TIBCO LogLogic Support, visit this site:

https://support.tibco.com/loglogic.htm

Entry to this site requires a username and password. If you do not have a username, you can requestone.

http://www.tibcommunity.com

http://docs.tibco.com

http://www.tibco.com/services/support

https://support.tibco.com/loglogic.htm

Universal Collector

Chapter

1Introduction

UC collects the information from four types of log sources: Syslog, Windows Event Logs, Real-Time File pull,or Remote File pull. Several UC agents can be deployed on a dedicated/shared appliance or physical/virtualhardware to remotely collect hundreds of log sources located at the same site.

Topics

• Overview

Overview

UC can seamlessly collect and forward logs to multiple log sources.

Collecting Logs — UC allows you to gather data from several types of log sources while ensuring integrityof the logs. You can easily collect event logs from local or remote instances of MS Windows, includingtime-stamped or rotated files. The UC agent works as a Syslog listener.

Forwarding Logs — UC has the ability to forward secure and authenticated data to an LMI server via aULDP protocol without the need for a dedicated appliance. UC also forwards to Syslog server using eitherUDP or TCP protocols.

Monitoring Activity — A UCMon tool is also available to monitor the internal process of the UC whichensures that your collection and forwarding processes are correctly responding.

Easy Configuration — A UC configuration is composed of Log Sources, Forwarding connections, andUC general parameters. UC configuration must be created and updated via the GUI or the Command LineInterface. You can create, save, and store a configuration.

A stored configuration is useful:

• to create a configuration and then activate it whenever you want, even if an active configuration is open,i.e. another configuration is running on the system.

• to create several configurations and deploy them rapidly on other UCs.

Easy Management — Multiple UCs can be remotely managed using TIBCO LogLogic ManagementCenter (MC) and MC Agent configured and running on each UC Asset. MC is a software solution thatallows you to manage Assets, schedule batch upgrade for Assets, monitor system health check, andbackup and restore Asset data.

Adaptability — UC is a software program with a small footprint and low memory usage on your DomainControllers, or application servers. It is highly adaptable and can be customized easily. Its lightweight andreliable configuration helps you to manage changes according to your particular needs.

Overview | 12

Universal Collector

Universal Collector

Chapter

2Collecting Logs

UC handles file collection from four different types of files.

Topics

• Real-Time File Logs• Windows Event Logs• Syslog Logs• Remote Files• UC Internal Logs• Creating and Configuring Log Sources• Creating a Complete Configuration• Editing Log Sources• Sorting Log Sources

Real-Time File Logs

UC reads logs from local files - i.e. logs from files generated on the machine where UC is installed andforwards them to either an LMI or a Syslog server.

UC can collect single and multi-line messages.

Real-Time File Logs | 14

Universal Collector

Collecting Single-line Messages

When a file is collected, only the newly added logs at the end of the file are collected. Logs alreadyavailable in a file before the UC log source creation will not be collected.

UC operates by monitoring specified text files that are receiving log output from log sources. The logsources append new logs to the end of the text file as events occur.

As new records appear at the tail of the monitored file, they are instantly taken into account by UC.

UC forwards single-line log messages to an LMI or Syslog server. By default, UC sends a maximum of64000 characters per line.

UC uses cursors to track the monitored files and to resume continuously after having stopped.

The cursors have information about the file positions at which to restart - called metadata - as well as fileidentification information.

It can determine whether the file to be resumed is the file to which the saved position applies.

In other terms, even if the UC is stopped for a while, all messages contained in the file will be collectedusing the position cursors, no messages will be lost.

Log File Rotation

In the case of log file rotation, a log file is retired and renamed to a “rotated” name, and the monitored file isreplaced by a new log file. Therefore, periodically during the monitoring of a log file that is rotated, the file isreplaced by a fresh log file.

UC is able to manage rotation files in two different ways.


Universal Collector

1. The log file name contains a date that changes during the rotation process

UC handles the rotation process of logs that contain a date in their name provided you correctlyconfigured the File Log Source configuration file.

If you enter the parameter [date] in the file path you must:

a. Activate the file rotation.

Enable and enter a date format for the date pattern such as yyyyMMdd.

For example,

Filenames: logFile.20110521.log, logFile.20110522.log

Absolute path: c:\logDir\logFile.[date].log2. The log file name contains an id that changes during the rotation

UC handles the rotation process of logs that contain an Id in their name provided you correctlyconfigured the File Log Source configuration file.

If you enter the parameter [id] in the file path you must:

1. Activate the file rotation.2. Enable and enter the number of digits expected (1-9) for the nbDigit parameter.

For example,

Filenames: logFile.1.log, logFile.2.log

Absolute path: c:\logDir\logFile.[id].log

You can combine the two examples to allow the use of both [id] and [date] parameters in the filepath.

Recommendations

• In the case of resuming after having been stopped, if the log file has been rotated during the period inwhich the collector was stopped, some log data will be missed. Therefore, you must ensure that thecollector is not temporarily stopped during an interval in which a rotation occurs.

• To be collected, a file must have been modified after the latest collected file.• The log file name does not change during the rotation. The UC records the “identity” of a log file in the

cursor as a hash of the first several bytes of the file. When the file is rotated and replaced with a freshone, the hash will be different. File identity checking is performed throughout the log file monitoringprocess to detect log rotation.

• If a log file needs to be replaced and enriched while UC is running, do not copy content in the file butmove it on the same partition.


Universal Collector

Collecting Multi-line Messages

UC can combine multiple consecutive related lines or multi-lines in a source log file into a single line whichwill be sent to the LMI. Multi-line message groups may require analysis to determine the correct expressionto use if the format is complex. UC supports Java regular expressions.

Before sending, groups of lines that represent a logical message are converted to a single-line format. Allof the original messages' data is kept intact – nothing is altered.

UC can collect multi-line messages from default application sources or custom ones:

Log Source Description

Tomcat / Servlet Container Default log location is CATALINA_BASE/logs. Tomcat and application logsunless configured otherwise. The default format is multi-line, with the firstline beginning with a timestamp. It may change due to localization. Logsare rotated daily by default

WebLogic Application Server Default log location is under the server root DOMAIN_NAME/servers/ADMIN_SERVER_NAME/logs/. Each server or cluster maintains a server log andselected events are forwarded to a domain log. Most of the entries aresingle line, but can contain java exceptions. Each message begins with'####'. There may also be a web access log

WebSphere ApplicationServer

Default log location is under the WebSphere directory APPSERVER/profiles/PROFILENAME/logs/SERVERNAME/. There is no default log rotation. There areserver start and stop logs (SystemErr.log, SystemOut.log), JVM log files(native_stderr.log, native_stdout.log), and process log files (startServer.log,stopServer.log). All of these logs contain entries describing the systemenvironment that do not have a timestamp. The error logs do not containany timestamps. Continuation lines are indented

JBoss Application Server Default log location is JBOSS_HOME/server/NAME/log. The boot log recordsstartup events prior to the initialization of the logging service. Theserver.log file records activity while the server is running. The boot.log fileentries begin with a time with no date. The server.log file entries start witha timestamp in the form 'YYYY-MM-DD HH:MI:SS,FFF'. Log messagescan be multi-line and the continuation lines are sometimes indented, butfrequently not. Messages start with a timestamp.

Note: The regex format for these default applications are indicated in <InstallationFolder>\runtime\conf\static\line_combiner.xml file.

Custom multi-line Custom regex can be defined for custom multiline logs. You need to define

- the header regex pattern.

- whether you keep orphaned lines, i.e UC sends messages that do notmatch the Header Regexp

- the timeout after which messages are sent even if the regex is not foundagain.

Custom Multi-line Log Sources

Custom regex can be defined for custom multiline logs.

• the header regex pattern.

Windows Event Logs | 17

Universal Collector

• whether you keep orphaned lines, i.e UC sends messages that do not match the Header Regexp• the timeout after which messages are sent even if the regex is not found again.

An example of a custom application log is as follows:

2010-03-19 16:09:41,344 WARN [main] file.FileImportSqlDao (?(think)) - File not found (/home/exaprotect/

conf/TBSMP6/report/etc/export.properties)

2010-03-19 16:09:41,344 WARN [main] config.ConfigurationFactory (ConfigurationFactory.java:127) - No

configuration found. Configuring ehcache from ehcache-failsafe.xml found in the classpath: jar:file:/

home/exaprotect/report_TBSMP6/webapps/ExaReport/WEB-INF/lib/ehcache-1.2.2.jar!/ehcache-failsafe.xml

java version "1.6.0_18"

Java(TM) SE Runtime Environment (build 1.6.0_18-b07)

2010-03-19 16:09:50,723 INFO [main] config.FacesConfigurator (FacesConfigurator.java:151) - Reading

standard config org/apache/myfaces/resource/standard-faces-config.xml

Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)

In the UC Console, you can create a regex like:

^\d+-\d+-\d+\s\d+:\d+:\d+,\d+\s[^\s]+\s+\[[^\]]+\]\s

with a timeout of 3 seconds and indicating that orphaned lines are kept.

It will match the header of the multiline log (date and level), which is:

2010-03-19 16:09:41,344 WARN [main]

All the lines will be aggregated and then forwarded as a single log to LMI. The \r and \n will be replaced byescaped ones \\r and\\n), until UC finds another regex header.

You can obtain something like:

2010-03-19 16:09:41,344 WARN [main] file.FileImportSqlDao (?(think)) - File not found (/home/exaprotect/

conf/TBSMP6/report/etc/export.properties)

2010-03-19 16:09:41,344 WARN [main] config.ConfigurationFactory (ConfigurationFactory.java:127) - No

configuration found. Configuring ehcache from ehcache-failsafe.xml found in the classpath: jar:file:/

home/exaprotect/report_TBSMP6/webapps/ExaReport/WEB-INF/lib/ehcache-1.2.2.jar!/ehcache-failsafe.xml\r

\njava version "1.6.0_18"\r\nJava(TM) SE Runtime Environment (build 1.6.0_18-b07)

2010-03-19 16:09:50,723 INFO [main] config.FacesConfigurator (FacesConfigurator.java:151) - Reading

standard config org/apache/myfaces/resource/standard-faces-config.xml\r\n

Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)

Refer to Appendix to get the full content of the Real-Time File Log Source commented file.

Windows Event Logs

UC can collect Windows Event Logs on Windows systems. However, it is not supported on Linux systems.

The supported Windows versions for remote collection are Windows 2003 R2 (32/64-bit), Windows 2008(32/64-bit), Windows 2008 R2 (64-bit), Windows 7 (32/64-bit), and Windows 2012 (64-bit).

UC forwards Windows logs to the LMI appliance by using the ULDP. Windows logs collected fromUC are forwarded in a format which is based upon the SNARE format. Although UC and SNAREformats are not 100% similar, a subtle difference may exist for certain messages.


Universal Collector

Non-administrator user accounts s can collect Windows Event Logs from remote event host. Foradministrator user accounts, UC auto-discovers the platform family and language type of the remote eventhost. For non-administrator user accounts, you should manually set the platform and language type oneach Windows event host using the advanced option and must set the following configuration settings:

• Enable the Remote Registry Service on the remote event host• On Windows 2008, Windows 7, and Windows 2012 Domain Controller systems, the non-administrator

domain user must be created and added to the Event Log Readers Group. On domain membersystems, the local user must be created on each local host and added to the local Event Log ReadersGroup. However, the domain user created on the Domain Controller system will not be able to accessthe event logs on the domain member system. On Windows 2003, refer to:http://blogs.technet.com/b/janelewis/archive/2010/04/30/giving-non-administrators-permission-to-read-event-logs-windows-2003-and-windows-2008.aspx.

http://blogs.technet.com/b/janelewis/archive/2010/04/30/giving-non-administrators-permission-to-read-event-logs-windows-2003-and-windows-2008.aspx




Universal Collector

Local CollectionThis section explains how to prepare a Windows host for local collection.

Enable the following Windows services:

• Windows Management Instrumentation• (For Windows 2003 only) Remote Registry


Universal Collector

Remote CollectionThis section explains how to prepare a Windows host for remote collection.

Enable the following Windows services:

• Windows Management Instrumentation• (For Windows 2003 only) Remote Registry

If Windows Firewall is enabled, run the following command to enable access to the above services:

netsh firewall set service RemoteAdmin enable

Syslog Logs | 21

Universal Collector

Filtering Windows Event LogsIt may be required to minimize Windows Audit events generated by certain UC activities via one of thefollowing methods:

1. Removal of “Object Access/ Success” from the audit policy on Windows log sources. (For furtherdetails, reference Audit Policy Management on Windows below.)

2. Review the current Security Access Control List (SACL) settings for the Windows Event Logsnamespace \\root\CIMV2, and verify that Enable Account/Successful is not checked for accounts/group to which the UC is connected. If necessary, create a new policy for the UC for which the EnableAccount/Successful is not checked.

If necessary, inheritance of SACL may have to be disabled for that namespace.

Platform Description

Windows 2003 R2/Windows 2008

The audit policy in Windows is configured via local policies and/or GPOlinked to domain/OU/Site. A good way to understand the resulting policy isto use ‘Resulting set of policy’ snap-in of MMC. Check the current resultingpolicy is set to generate results for local host only. The current resultingpolicy can be found under Computer Configuration > Windows Settings> Local Policies > Audit Policy.

Windows 2008 only On Windows 2008 more granular settings are possible, named “sub-category”. Based on the solution used, you can check the precise auditingpolicy with: auditpol /get /category:*

For more information on sub-category audit capabilities, please refer to theMicrosoft documentation.

http://support.microsoft.com/kb/921468


Also review the article on Windows Event Logs namespaces mentioningspecifically Windows Event Logs auditing:

http://msdn.microsoft.com/en-us/library/aa822575(v=vs.85).aspx

Syslog Logs

UC reads logs sent via the Syslog protocol. The syslog logs will be collected using the TCP or UDP.

UC will not start up a syslog listener on the desired port until at least one syslog collector exists.

If you want to use both protocols, you must define two Log Sources.

Protocol Description

UDP Default configuration. It specifies that the syslog logs should be collectedvia UDP protocol. When modifying the UC’s status (such as updating orstopping it) or when the UC is not running during the collection, messages



http://msdn.microsoft.com/en-us/library/aa822575(v=vs.85).aspx

Syslog Logs | 22

Universal Collector

Protocol Description

may be lost. Indeed, contrary to the TCP protocol, the UDP protocol avoidsthe overhead of checking whether every packet actually arrived, which maylead to data loss.

TCP Specify that the syslog logs should be collected via TCP protocol. If anotherSyslog log is running on the server where the UC is installed, the UC andsyslog will not have the same port, IP and protocols. In that case, you musteither stop the syslog or make the UC listen on another port.

Remote Files | 23

Universal Collector

Filtering Syslog Logs

The Syslog logs can be filtered, before being forwarded, according to their severity and facility.

• facility - type of message that must be collected.• severity - levels of severity that will be reported.

If a message has neither severity nor facility, UC automatically allocates the local use 7 facility and thedebug severity to the message. Then, it will be automatically filtered.

Remote Files

UC can collect files remotely and forwards them to LMI.

By default, UC pulls every 1 hour but it can also pull every X minutes, every X hour, daily at X time, weeklyon Y day on X time.

It is highly recommended to use a physical machine for remote file collection. It is notrecommended to use the UC's remote file collection to collect large remote files (above 1GB) onVirtual Machine systems as it will slow down the system significantly.

Remote File with Rotation

In the case of log file rotation, a log file is retired and renamed to a “rotated” name, and the monitored file isreplaced by a new log file. Therefore, periodically during the monitoring of a log file that is rotated, the file isreplaced by a new log file.

When the date field is checked for rotation, UC will only collect files that are modified after the remote filelog source creation time.

UC is able to manage file rotation in two different ways. For more information, refer to Log File Rotation onpage 14.

Remote File with No Rotation

• Single Files

Make sure that you gave the correct file path on the remote file system to pull the file correctly.• Directory

Directory pull allows you to choose a directory and pull files from that directory based on the ‘include’ or‘exclude’ options provided to you. Directory pull does not support file rotation.

Example: /loglogic/ directory has three files: a.txt, b.txt, c.txt

Scenario 1: if users put * for include, it will pull a.txt, b.txt, c.txt

Scenario 2: if users put *.txt for include and put a.txt for exclude, it will pull b.txt and c.txt

Scenario 3: if users put a.txt for include and nothing for exclude, it will only pull a.txt.

UC Internal Logs

UC generates its own logs when it is subjected to changes or errors (for example, starting of the UC,creation of a Log Source, disconnection of the UC, etc.). These internal logs are also sent to the LMI andcan be used to repair or troubleshoot the UC.

Creating and Configuring Log Sources | 24

Universal Collector

Collecting UC Internal Logs

The UC internal logs are automatically generated in the uc.log file, which is located in the UC installationfolder in \LogLogic\UniversalCollector\logs (for Windows).

The uc.log is forwarded to the LMI provided you correctly configured the forwarding process (LMIconnection).

The LMI connection used to forward the UC internal logs can be the same as any log source LMIconnection.

Creating and Configuring Log Sources

You can add, copy, and delete Log Sources.


Universal Collector

Add a New Log SourceYou can add a new Log Source.

1. Open the UC Console by clicking on the shortcut and click the Collection tab.

2. Click New and select the type of Log Sources you want to add; Real Time File, Syslog, Windows EventLog or Remote Files.

3. In the Edition screen, enter the relevant information as explained in Editing Log Sources on page34.

4. Click Save to save the Log Source.A new log source is added in the list of Log Sources.


Universal Collector

Copy a Log SourceYou can copy one or multiple Log Source configurations.


2. Select one or several Log Sources (Ctrl + click to select more than one Log Source) from the list of logsources.

3. Click Copy and confirm.The new log source(s) is/are displayed below the list of log sources. You can edit and modify as anyother log source.

By default, the log source configuration is not enabled.


Universal Collector

Delete a Log SourceYou can delete one or multiple log sources.


2. Select one or several Log Sources (Ctrl + click to select more than one Log Source) from the list of logsources.

3. Click Delete.The Log Source list is automatically refreshed.


Universal Collector

Creating Multiple Log SourcesYou can import and create multiple Log Sources of the same type at the same time.

Make sure that a CSV file with Log Source information must be available.

Create a CSV File

1. Open a program such as Notepad.

2. In the header, on the first line, enter the following field names according to the type of Log Source youwant to create:

Log Source Fields

File name, description, lmi_connection*, enabled, timeInUtc,message_filter, match_filter, file_path*, useDateRolling,date_pattern, useIdRolling, nbDigit, useFileChangeNotification,multiline_active, multiline_header_type, multiline_custom_regex,multiline_orphaned_lines, multiline_lineTimeout, appName*, hostname*,maxLineLength, charset

Syslog name, description, lmi_connection*, enabled, timeInUtc, protocol, ip,port, severity, facilities, source_ip

Windows name, description, lmi_connection*, enabled, timeInUtc, event_id_filter,filter_operator, source_filter, address*, domain, login, password,include_eventlogs, eventlogs_list, polling_period, win_type, lang_type

Remote File name, description, enabled, lmi_connection, ip,protocol, time_zone,file_system_type, user_id, password, domain, share_name, path_type,path,original_name, include, exclude, useDateRolling, date_pattern,useIdRolling, nbDigit, useFileChangeNotification, useUcIP, uc_ip,every_minutes, every_hours, daily_at_time, weekly_at_time,weekly_at_day, device_type

* mandatory fields

Creating a Complete Configuration | 29

Universal Collector

Log Source Fields

1.LMI connection is mandatory only if there is more than one existing connections available.The sole connection will be taken by default. 2. Name is not mandatory as a name will beautomatically created, such as Real Time File #n or Windows Event Log #n or Syslog #n.

3. On the lines below, fill in the fields with the correct values and save in CSV format.The CSV file format example:

name,description,lmi_connection,timeInUtc

Log Source A, Windows Log Sources, LMI_Connection, true

A detailed example of the fields and values to enter in the CSV file is available from UC Consolewhen importing the CSV file.

Import Log Sources

1. Open the UC Console by clicking on the shortcut.

2. In the Collection tab, click New > Batch import.The Batch Import tab is displayed.

3. In the drop-down list, select the type of Log Sources you are going to import.

4. Browse the CSV file and click OK.

5. Click Import.The Log Sources are created under the Collection tab, for example, Import #1 - LS #1

Creating a Complete Configuration

A configuration contains general settings, a list of Log Sources, and one or several Forwardingconnections.

All of these items are configured via the Graphical User Interface and are stored in a UC Configuration file(*.ucc) that you can unzip to explore the content.


Universal Collector

Edit Configuration General SettingsYou can modify the default configuration at any time.


2.Click .

3. Modify the following information:

Option Description

Name Name of the configuration.

Communication Port Port used by the UC to get information (for example, status, metrics,memory used...) via the CLI. Make sure this port is not already used.Otherwise UC will not work.

Collector Domain An identification name used to identify each message sent from aspecific UC. This field can be empty. If defined, it must have a uniquename with maximum 256 characters. This field is case sensitive. Donot include special characters, for example, \|/"?'*:%

TCP/UDP socket buffer size TCP/UDP parameter and socket buffer size (in kilobytes) - thisparameter applies to all the Syslog Log Sources associated to the UC.

UDP max packet size UDP parameter and max packet size (in kilobytes) - this parameterapplies to all the Syslog Log Sources associated to the UC. Themaximum size is 64KB.

Notes for Red Hat and SUSE Linux Enterprise

If you obtain a log message saying “Syslog Unable to set the required socket buffer size”, then it isrecommended to increase the maximum size of the buffer on your RHEL, SUSE, and Solaris systems.

On RHEL, SUSE or Solaris, the default maximum TCP/UDP buffer size is 128 KB.

In the UC configuration file, the default value of the buffer socket size is 1MB. These parameters applyto all the Syslog Log Sources related to UC. Therefore, you must increase the maximum value of theSyslog buffer already set with a specific command.

To change the maximum value of the buffer:

1. Log in as root on the system.2. Enter the following command (example with 1 Megabyte):

sysctl -w net.core.rmem_max=1048576 (this value is expressed in bytes)

The modification of the system parameter will impact the maximum limitations for all sockets.

4. Click Apply.The configuration is updated.


Universal Collector

Add a New ConfigurationYou can easily add a new configuration. After adding a new configuration, you must activate it.


2. Go to Manage Configuration > New.

3. In the Browsing window, select a folder where you will store your configuration.

4. Enter a configuration name with a *.ucc extension in the Filename field and click Save.The new configuration is automatically displayed in the UC console, but it is not active.


Universal Collector

Open a Stored ConfigurationYou can edit an existing or stored configuration other than the one running on the local UC at any time.


2. Under Manage Configuration, click Open and browse the UC configuration file (*.ucc).

3. Click Open.The configuration is displayed in the GUI. However, this configuration is neither applied nor running.

You can display back your active configuration at any time by selecting Manage Configuration> Open Active Configuration in the drop-down menu.


Universal Collector

Activate the ConfigurationYou can make a stored configuration active at any time. Then, all the modifications applied on the freshactive configuration will be automatically saved and updated each time you validate the changes.


2. Display the configuration that you want to activate in the UC Console.

3.Click .A warning message is displayed which indicates that the active configuration will be overwritten if youcontinue.

4. Click Continue to accept. If you do not want the active configuration to be erased, click Cancel andmake a copy of it before activating another configuration.The configuration is now active and can be modified.

Editing Log Sources | 34

Universal Collector

Save a ConfigurationYou should save an active or stored configuration on the local system.


2. To copy a configuration, select Manage Configuration > Save as.

3. In the Browsing window, select the folder where you want to save the configuration. You can create anew folder.

4. Name the configuration and click Save.A UC Configuration file with the *.ucc extension is created.

Editing Log Sources

You can edit a single Log Source configuration. Similarly, you can update parameters for multiple LogSources of the same type at a time.


Universal Collector

Edit a Real-Time File Log Source

1. Under the Collection tab, double-click on the selected Log Source or just select it and click the Editbutton.The RT File Edition tab is displayed.

2. In the General part of the screen, you can modify the following information:

Option Description

Log Source Enabled Click ON or OFF to define whether the current Log Source is enabled ordisabled.

Name Name of the Log Source.

Description Description of the Log Source.

3. In the Forwarding Connection part of the screen, you can modify the following information:

Option Description

Name Select the Forwarding connection to which you want to forward collected RTFile logs. A Log Source must be linked to an existing Forwarding connection,which can be edited under the Forwarding tab.

UC Collection date Define whether the log message sent to the LMI server remains in a localsystem time zone or is converted into UTC time zone.

4. In the Message Filtering part of the screen, you can modify the following information:

UC supports Java regular expressions.

Option Description

[Filtering] Click ON or OFF to activate or deactivate the option.

Collect messages Define whether you collect messages that:

- match the regex (other logs are filtered)

- do not match the regex (i.e. filter the logs that match the regex)

Filter Enter a case insensitive regular expression to specify the messages to bematched.

For example, if “Not matching regex is selected”:

"packet accepted" means that all the lines containing packet accepted arefiltered.

"^64\.242" means that all the lines that are beginning exactly with 64.242 arefiltered

"846$" means that all the lines that are ending exactly with 846 are filtered.

For example, if “Matching regex is selected”:

"packet accepted" means that only the lines containing packet accepted arekept.


Universal Collector

Option Description

"^64\.242" means that only the lines that are beginning exactly with 64.242are kept.

"846$" means that only the lines that are ending exactly with 846 are kept.

5. In the Collection part of the screen, you can modify the following information:

On Windows, Real-Time file collection is unavailable on network shared and Network FileSystem (NFS) mounted drives.

Option Description

File Path Browse the log file to be collected.

If the log file is rotated, you may enter [id] or [date] or both in the filename aswell as configuring the File rotation parameters.

For example, c:\temp\logFile[date].log to obtain file names such aslogFile20110521.log

For example, c:\temp\logFile[id].log to obtain file names such aslogFile1.log

File rotation Click ON or OFF to activate or deactivate the option.

[If File rotation is ON]Date pattern

Enter the date format you want to use for the [date] parameter.

For example, yyyyMMdd for 20120421.

[If File rotation is ON]Max number of digits

Check the box and indicate the maximum number of digits you want for the[id] parameter.

UC can collect any file with an [id] whose number of digits is between 1 and9 inclusive.

For example, If you set 5, the following [id] will be taken into account: 1, 054,586, 00599, 78945, etc.

File change notification Click ON or OFF to activate or deactivate the option. This option allows youto monitor file changes. If set ON, a notification will be sent to LMI via uc.logfile when the specified file's modified date changes. The notification includesthe changed content and time. A new log is recorded for the notificationwhen UC internal logs are forwarded to LMI. The file changes are notmonitored for rotated files. In this case, the File change notification option isdisabled.

The specified file size should be less than the default size (10MB). If the filesize is more than 10MB, the notification does not include changed content.

Before activating this monitoring option, make sure to set the LMIConnection > Forwarding > Forward UC Internal Logs option toON.

[Multiline messages] Click ON or OFF to activate or deactivate the option to define whether thesingle message has several lines.


Universal Collector

Option Description

[If Multiline messagesis ON] Multiline headertype

Select the type of multi-line logs.

For example, 'jboss', 'tomcat', 'weblogic', 'websphere' or 'custom'.

[If Multiline messagesis ON] Custom headerregex

Set a regular expression matching the header of the first line of a log.

[If Multiline messagesis ON] Send orphanedlines

Indicate whether you want the UC to send messages that do not match theHeader Regexp.

[If Multiline messagesis ON] Multiline timeoutafter detected header

Indicate the number of seconds after which the multi-line logs are ready to besent.

[Advanced] Click the drop-down menu to display advanced parameters.

Host name Enter the name of the host used to pair logs on the LMI server.

For example, customHostname.com

If you enter an IP address, the device to be displayed in LMI will be referredwith this IP address.

Application name Enter the name of the application used to identify logs on the LMI server.

For example, customApplicationName

Maximum messageslength

Indicate the possible maximum length for the message (in bytes).

Default value: 64000

[Collected file]

Charset

Select the data format.

Default value: Use local system charset

6. Click Apply to validate the changes.


Universal Collector

Edit Multiple Real-Time Log Sources

1. Under the Collection tab, select the Log Sources and click the Edit button.The RT File Edition tab is displayed.

2. Check the boxes in front of the set of RT File parameters you want to change.

3. Modify the parameters as explained in Edit a Real-Time File Log Source on page 35.


Universal Collector

Edit a Windows Event Log Source

1. Under the Collection tab, double-click on the selected Log Source or just select it and click the Editbutton.The Windows Events Log Source Edition tab is displayed.


Option Description



For example, ls-win-template



Option Description




Option Description


Event ID Filter Regular expression to filter the Windows event ID.

For example,

“567|^58[1-9]” means that the events with an Event ID containing 567 butalso those from 581 to 589 inclusive are collected.

“^(8.*)|^(5[2-9].*)” means that the events with an ID starting with 8 but alsothose starting with 52 to 59 inclusive are collected.

If the field is empty or .* is set means that no filter is set.

Refer to Regular Expressions on page 111 to get the list of charactersused in regular expressions.

and/or Select if you want to use both filters at the same time or one or another

Source Filter Enter a regular expression to filter Windows events on source field.

For example,

“Security” means that all the events with a Security source field are filtered.

“DNS Client Events” means that all the events with a DNS Client Eventssource field are filtered.


Universal Collector

Option Description

“Time-Service” means that all the events with a time-Service source field arefiltered.

If the field is empty or .* is set means that no filter is set.

Refer to Regular Expressions on page 111 to get the list of charactersused in regular expressions.


Option Description

[Location]

Local/Remote host Indicate whether the Windows host from which to poll logs is the localmachine or a remote host.

Host name Enter the IP address to connect to the remote Windows server.

[Credentials]

Use UC servicecredentials/Use customcredentials

Select the relevant options to use the correct Windows credentials.

If you have configured credentials in the UC Windows ServicesControl Panel, you can use those credentials to create multipleWindows Event Log Collections. To do this, select the UC servicecredentials option.

Domain (if Use customcredentials is set)

Enter the domain name to access the Windows server.

For example, domain.company

Login (if Use customcredentials is set)

Enter the login to connect to the Windows server. If the user has non-administrator privileges, make sure to satisfy the prerequisites specified inthe section Windows Event Logs on page 17.

If the login belongs to a local user with administrator privileges, theUser Account Control (UAC) needs to be turned off at the eventhost.

Password (if Usecustom credentials isset)

To connect to the Windows server, enter a password

[Windows Event Logs]

Collect Define the Windows Event Logs journals to include. It can be either:

- all event logs = all current and logs to come are collected

- all event logs except the following ones = all current and event logs tocome are collected except the one indicated in the List form.

- only the following event logs = only the following event logs indicated in theList form are collected

List List of Event Logs to include or exclude.

Edit List Displays the Edit List window to select the event logs to be collected:


Universal Collector

Option Description

1 - In the Available Event Logs pane, select an event log and click Add.This will add the logs to the list.

2 - If you want to remove them from the list, select them and click Remove.

3 - If you want to manually add an Event Log, enter the name and click Add.Make sure you entered the name correctly as it is case-sensitive.

4 - Click OK.

If you want to display all the Event Logs available, click the DiscoverEvent Logs button.

[Advanced]

Polling Period Enter the time period (in seconds) after which UC checks for new Windowsevents.

Default value: 10

Windows type Specify the platform from the drop-down list.

If you do not specify the platform type, UC will try to auto-discoverthe platform type. However, if the user has non-administratorprivileges, UC will fail to auto-discover the platform type.

Language type Specify the language type from the drop-down list.

If you do not specify the type, by default it will be assigned asEnglish.



Universal Collector

Edit Multiple Windows Event Log Sources

1. Under the Collection tab, select the Log Sources and click the Edit button.The Windows Event Log Edition tab is displayed.

2. Check the boxes in front of the set of Windows Event Logs parameters you want to change.

3. Modify the parameters as explained in Edit a Windows Event Log Source.


Universal Collector

Edit a Syslog Log Source

1. Under the Collection tab, double-click on the selected Log Source or just select it and click the Editbutton.The Syslog Log Source Edition tab is displayed.


Option Description





Option Description




Option Description

Protocol Define whether the Log Source uses the udp/tcp SYSLOG protocol.

In order to listen on both UDP and TCP protocols, you must create twoSyslog Log Sources.

Port Enter the port to listen to the Syslog flow.

Default value: 514

Binding interface If there are multiple network interfaces, enter the IP address to listen to theSyslog flow. Only one IP address is possible.

To listen to all network interfaces, use 0.0.0.0.

To listen to a specific interface, use an address like 192.168.11.10

Default value: 0.0.0.0


Option Description


If Message Filtering is set on OFF, messages with a ‘debug’ severity are notcollected (max severity set to 6).


Universal Collector

Option Description

If a message has neither severity nor facility, UC automatically allocatesthe local use 7 facility and the debug severity to the message. It will then beautomatically filtered.

Maximum Severity Select the maximum accepted severity (numerical code, see RFC 3164)

0 - Emergency: system is unusable

1 - Alert: action must be taken immediately

2 - Critical: critical conditions

3 - Error: error conditions

4 - Warning: warning conditions

5 - Notice: normal but significant condition

6 - Informational: informational messages

7 - Debug: debug-level messages

Default value: 6 - Informational: informational messages

Authorized facilities Select one or several accepted facilities (see RFC 3164). The logs with thesefacilitities are kept.

0 - kernel messages

1 - user-level messages

2 - mail system

3 - system daemons

4 - security/authorization messages (note 1)

5 - messages generated internally by syslogd

6 - line printer subsystem

7 - network news subsystem

8 - UUCP subsystem

9 - clock daemon (note 2)


11 - FTP daemon

12 - NTP subsystem

13 - log audit (note 1)

14 - log alert (note 1)


16 - local use 0 (local0)






Universal Collector

Option Description




Default value: 0-23

Authorized IPaddresses

Enter the regular expression to filter the accepted IP addresses and to filterthe accepted host.

All the logs from all IP addresses are collected if the field is blank (default).



Universal Collector

Edit Multiple Syslog Log Sources

1. Under the Collection tab, select the Log Sources and click the Edit button.The Syslog Log Source Edition tab is displayed.

2. Check the boxes in front of the set of Syslog parameters you want to change.

3. Modify the parameters as explained in Edit a Syslog Log Source on page 43.


Universal Collector

Edit a Remote File Log Source

1. Under the Collection tab, double-click on the selected Log Source or just select it and click the Editbutton.The Remote File Log Source Edition tab is displayed.


Option Description





Option Description


Remote File Collection is only supported by LMI v5.4 or above and can only be forwarded toLMI, not generic syslog servers.


Option Description

Host IP/Name Enter the IP or name of the remote log source.

Protocol Define whether the Log Source uses the ftp, sftp, cifs or file protocol.

On Windows, Remote file collection using file protocol is unavailableon network shared and Network File System (NFS) mounted drives.

[If ftp is selected]Server TimeZone

Select the timezone of the remote log source.

[If a non-local timezoneis selected] File SystemType

Select the file system type.

User ID Enter the User ID to connect to the remote log source.

[If cifs is selected]Domain/User name

Enter the domain or user name.

User password Enter the user password.

[If cifs is selected]Share name

Enter the cifs share name.

File / Directory Select the source of the collection, either a file or the content of a directory.


Universal Collector

Option Description

[If File is selected] Filepath

If File is selected, enter the file path. This is the absolute path of the filesystem where the UC is installed. For example, on Windows: d:\myFolder\myLog.log. However, on Linux/UNIX systems it must be as /usr/myAccount/myLog.log.

[If File is selected] FileRotation

Click ON or OFF to activate or deactivate the option.

Only available if File is selected.

[If File is selected] Filechange notification

Click ON or OFF to activate or deactivate the option. This option allowsyou to monitor file changes. If set ON, a notification will be sent to LMI viauc.log file when the specified file's modified date changes. The notificationincludes the changed content and time. A new log is recorded for thenotification when UC internal logs are forwarded to LMI. The file changesare not monitored for rotated files and directory. In this case, the File changenotification option is disabled.

The specified file size should be less than the default size (10MB). If the filesize is more than 10MB, the notification does not include changed content.

Before activating this monitoring option, make sure to set the LMIConnection > Forwarding > Forward UC Internal Logs option toON.

[If File Rotation is ON]Original name

The file that is currently being written; it is usually the file without date or idtag.

[If File Rotation is ON]Date pattern

Enter the date format you want to use for the [date] parameter.

For example, yyyyMMdd for 20120421

[If File Rotation is ON]Max number of digits

Check the box and indicate the maximum number of digits you want for the[id] parameter.

UC can collect any file with an [id] whose number of digits is between 1 and9 inclusive.

For example, If you set 5, the following [id] will be taken into account: 1, 054,586, 00599, 78945, etc.

[If Directory is selected]Directory path

If Directory is selected, enter the directory pathname.

[If Directory is selected]File(s) Include

Enter the files that must be included in the collection. The field supports thestandard common wildcard characters for matching file names (* and ?).

[If Directory is selected]File(s) Exclude

Enter the files that must be excluded from the collection. The field supportsthe standard common wildcard characters for matching file names (* and ?).

Device type Select the type of logs to be collected.

Test connection Click this button to check if the connection to the remote log source isworking.

[Advanced]

Log Source IP

Select an option:

- Remote file server: selected by default. The IP is grabbed from the host IPthat you previously entered.


Universal Collector

Option Description

This option is not available when the file protocol is selected.

- UC: IP address of the workstation where UC is installed. You can change itas you want.

The IP address will be set as the host IP address when the fileprotocol is selected.

[Schedule] Select the collection period, either per minute, hour, daily or weekly at aspecific hour.



Universal Collector

Edit Different Types of Log SourcesYou can edit several Log Sources of different types, except remote files, at a time. Only the commonparameters are editable.

1. Under the Collection tab, press Ctrl while clicking on the Log Sources to select them.

2. Click Select screen to only select the list of visible Log Sources at a time or click Select all to select allthe lists of Log Sources.

3. Click the Edit button and select All. The All tab is displayed.


Option Description





Option Description



6. Click OK to save the changes. If you open again one of the Log Source you selected, you can see thatthe changes are applied.

Sorting Log Sources | 51

Universal Collector

Sorting Log Sources

Tags are useful to store, sort, and search for Log Sources in a list.

For example, if you want to easily find the logs coming from Windows server A to which the administratorhas logged. You can create tags such as Server A, Connection, Administrator, and then search based ontags.

You can create and apply up to 10 filters.


Universal Collector

Create a New Tag

1. Under the Collection tab, select one or several log sources.

2. In the Tag edition panel on the right, enter a tag in the combo box and click Add Tag.The tag is automatically saved.


Universal Collector

Apply a TagOnce you have created tags, you can apply them to one or several log sources.

1. Under the Collection tab, select the log source(s) to which you want to apply a tag.

2. In the combo box in the right hand panel, select the tag you want to apply and click Add Tag.The tag is displayed under the Tags column.


Universal Collector

Remove a Tag

1. Under the Collection tab, select the log source for which you want to remove the tag.

2. In the Tag edition panel, click the cross of the tag you want to remove.The list is updated automatically.


Universal Collector

Sort Log SourcesYou can sort the list of log sources to display only the relevant items.

1. In the left hand part of the configuration panel, click the + Add Filter button.Two drop-down list boxes are displayed.

2. In the first drop down list, select the type of information you want to filter. The options are: Enabled,Name, Forwarder, Type, Collection or Tags.

3. Based on the type, select the relevant values.

Filter Values

Enabled Sorts log sources per status, i.e. Off or On.

Name Sorts log sources per name. Enter the log source name.

For example, ls-logsource-windows

Forwarder Sorts log sources per Forwarding connection (names of the connection file),for example, uldp-sample

Type Sorts log sources per type, i.e. file, syslog or windows.

Collection Sorts log sources per collection type, i.e. file, syslog or windows.

Tags Sorts log sources per user-created tags, for example, server, web.

4. Click Apply to filter the list.

5. To add another filter, click +Add Filter and repeat the procedure explained above. For example, tomake a search on a specific forwarder AND a specific type of file, you will obtain something like this:

6. For a same filter if you want to add another value, click the + button and select the relevant value. Forexample, to find a File Log Source OR a Syslog log source, you will have to obtain something like this:


Universal Collector

7. To remove a filter or only a value, click the - button.

8. Click the column header to display the filtered list by alphabetical order.

9. Click the Clear all button to disable the filters.

Universal Collector

Chapter

3Forwarding Logs

UC collects the information from various types of log sources and forwards them to an LMI server.

The logs are forwarded to an LMI server via the proprietary ULDP protocol or to a Syslog server using UDP orTCP protocols for the communication between the UC and the LMI server or syslog server.

You must select the UDP when forwarding syslog to LMI server.

A file is identified by a file identifier usually a string representing the path name of the file in the source device.

Topics

• Creating a Syslog TCP or UDP Connection• Creating an LMI Connection• Creating a Connection in Authentication and/or Encryption Mode• Managing the list of Forwardings

Creating a Syslog TCP or UDP Connection

You can add up to 10 Forwarding Connections.

1. Open the UC Console and click the Forwarding tab.

2. Select the New > TCP (Syslog) or UDP (Syslog) menu.

3. In the General section, modify the name of the connection.

4. In the Security section, make sure the button is set to OFF.

5. In the Forwarding section, modify the following values:

Forwarding

Address Enter the IPv4 address or host name of the TCP /UDP server.

Port Enter a port number. (Default: 514)

[TCP Only] TestConnection

Test the connection between UC and the server.

Message Format

Facility Select the facility to be applied to the log:

0 - kernel messages

1 - user-level messages

2 - mail system

3 - system daemons

Creating a Syslog TCP or UDP Connection | 58

Universal Collector


5 - messages generated internally by syslog

6 - line printer subsystem

7 - network news subsystem

8 - UUCP subsystem



11 - FTP daemon

12 - NTP subsystem

13 - log audit (note 1)

14 - log alert (note 1)










Severity Select the severity to be applied to the log:

0 - Emergency: system is unusable

1 - Alert: action must be taken immediately.

2 - Critical: critical conditions.

3- Error: error conditions.

4 - Warning: warning conditions.

5 - Notice: normal but significant condition.

6 - Informational: informational messages.

7 - Debug: debug-level messages.

Custom Header Indicate the header of the message.

Advanced

[TCP only] Sessiontimeout

Enter the session timeout (in seconds)

UC Binding interface If there are multiple network interfaces, enter the IP address that the UCuses when establishing the connection.

Default: 0.0.0.0

Creating an LMI Connection | 59

Universal Collector

6. In the Message Buffering section, modify the following values:

Message Buffering

Buffer size (MB) Enter the buffer size in megabytes. (Default: 100 MB)

7. Click OK to save and close the screen.The list of connections is updated.

Creating an LMI Connection


2. Select the New > ULDP to open the LMI Connection tab.

3. In the General section, modify the name of the connection.

4. In the Security section, make sure the button is set to OFF.

5. In the Forwarding section, modify the following values:

Forwarding

Address Enter the IPv4 address or host name of the LMI.

Port Select the LMI port or enter a port.

- 5514 for connection with LMI 5.0 and 5.1 (default value)

- 5515 for secured connection with LMI 5.0 or later (configurable in LMI)

- 5516 for connection with LMI 5.2 or later

Test connection Test the connection between UC and LMI.

Forward UC Internal Logs Define whether the UC internal logs are sent to the remote LMI byselecting ON.

Compress Messages If the connection is slow, you can configure the logs to be compressedfor a more rapid flow of data. Define whether the logs are compressed byselecting ON.

Advanced

Reconnection Enter the reconnection frequency to the LMI (in seconds)

Session timeout Enter the session timeout to LMI (in seconds)

UC Binding interface If there are multiple network interfaces, enter the IP address that the UCuses when establishing the connection to LMI.

Default: 0.0.0.0

6. In the Message Buffering part of the screen, modify the following values:

Message Buffering


Scheduled Forwarding Define the period of time during which the logs are sent to the LMI (timewindow) by selecting ON.

Creating a Connection in Authentication and/or Encryption Mode | 60

Universal Collector

Schedule forwarding is not recommended for pulling large filesvia remote file collection.

Daily Start Define the beginning of the time window. If sendingWindow = true inthe above parameter, define the time (hour and minute) when the eventstarts to be sent (default value = 23:00)

Daily Stop Define the end of the time window. If you set sendingWindow = true inthe above parameter, define the time (hour and minute) when the eventstops to be sent (default value = 05:00).

7. Click OK to save and close the screen.The list of LMI connections is updated.

Creating a Connection in Authentication and/orEncryption Mode

The information is delivered through the communication between the UC and LMI server or syslog servercan be encrypted.

To secure communications between the UC and LMI or syslog servers, the following information will bechecked: LMI or a syslog server and UC identities and encryption of communication between UC and LMIor a syslog server (public and private key mechanism).

If you need to use AES192 or AES256 key, you must install the Java Cryptography Extension(JCE) Unlimited Strength Jurisdiction Policy Files 7 package from Oracle. The 2 JAR files includedin this package must be loaded into the lib/security directory of the Java instance that UC usesin order to utilize AES192 or AES256 key ciphers. If you do not have JCE installed, then the UCConsole will fail when you try to import an AES192 or AES256 key.

As a requirement, you need a PKI and OpenSSL or another compatible tool.

This section is intended for advanced users with the necessary encryption and securecommunication skills.


Universal Collector

1. A public key and a private key are used to create a Root Certificate Authority (Root CA).

2. A public key and a private key are generated to create the UC’s Certificate Signing Request (CSR).

3. This request will be sent along with the UC’s identity information and the public key.

4. The Root CA delivers the certificate by signing the Certificate Signing Request. The UC’s certificate isthen created and sent with the Authority’s certificate.


Universal Collector

Step 1: Get a Root Certificate Authority from your PKI

When deploying an authentication process with UC, you need to use a Public Key Infrastructure (PKI)consisting of a certificate authority or CA (and a registration authority or RA) that issues and verifies digitalcertificate. A certificate includes the public key; one or more directories where the certificates (with theirpublic keys) are held and a certificate management system.

A number of products that enable a company or group of companies to implement a PKI exist.

1. Access a tool such as OPENSSL.

2. Generate a public and a private key. The recommended and maximum size is 2048 bit and encrypted inAES 128 (3DES is also supported).

Example: openssl genrsa -out ca.key -aes128 2048

3. Generate the CA (valid for 7305 days)Example: openssl req -new -x509 -days 7305 -key ca.key -out ca.pem

Refer to the SSL Certificates HOWTO documentation to know how to create your Certificate Authority:

http://www.gtlib.gatech.edu/pub/linux/docs/HOWTO/other-formats/html_single/SSL-Certificates-HOWTO.html




Universal Collector

Step 2: Create a Certificate Signing Request

You must now generate a Certificate Signing Request in a UC to be able to create a Certificate on aCertificate Authority. You will obtain a file with the *.csr extension.

Using the Internal ToolThe tool is located in <INSTALL_DIR>/tools folder.

1. Enter the following command to start the tool:

Windows:cert_mgt.bat

RHEL, SUSE, Solaris:cert_mgt

2. Enter the following command:<script-name> request

3. Enter the command to indicate the file path of the file to be generated. You have three possibilitiesaccording to the type of your certificates.

[ -jks <file path of the generated *.ks containing the private key> ]

[ -p12 <file path of the generated *.p12 certificate containing the private key> ]

[ -pem <file path of the generated *.pem private key> ]

-csr <file path of the generated Certificate Signing Request>

[ -dn <CSR Distinguished Name> ]

-pwd <mandatory password for the file containing the private key>

This command generates 2 files containing the private key (i.e. a *.ks or *.p12 or *.pem) and aCertificate Signing Request (CSR).

If it is not specified in the command line, by default, the DN of the CSR is:

CN=<UC-IP>, O=loglogic

For example: cert_mgt request -jks uc.ks -pwd loglogic -csr uc.csr

Using the OpenSSLYou need UC's public and private keys and OpenSSL.


Universal Collector

1. Generate the public and private keys. The recommended and maximum size is 2048 bit and encryptedin AES 128 (3DES is also supported):openssl genrsa -out uc.key -aes128 2048

2. Create the CSR like:openssl req -new -key uc.key -out uc.csr

Refer to the SSL Certificates HOWTO documentation to know how to create your Certificate Authority.


Universal Collector

Step 3: Create a Valid UC Certificate using a CA and OpenSSLYou must create the valid Certificate issued by a Certificate Authority in the UC configuration.

Enter the following command:openssl ca -config ”conf_file.txt” -days 730 -in uc.csr -out uc.pem -notext

In this example, a file has been defined (conf_file.txt). If no configuration file has beenspecified, then OpenSSL takes /usr/local/ssl/openssl.cnf by default.

You will get a *.pem certificate that contains the UC’s certificate.


Universal Collector

Step 4: Import the Certificate into *.ks or *.p12This step is not required if you work with a *.pem certificate.

This command allows you to import the UC certificate and/or the root CA certificate in a *.ks or the UCcertificate in a *.p12 certificate.

Using the CLI provided by LogLogic, enter the command to format the file:

<script-name> import

[ -jks <file path of the *.ks> ]

[ -p12 <file path of the *.p12 certificate> ]

-pwd <mandatory password>

[ -cert <file path of the UC certificate in *.pem format> ]

[ -rootcert <file path of the root CA certificate in *.pem format> ]

This command imports the UC certificate and/or the root CA.

You can obtain a *.ks certificate file that contains a Certificate Authority, private key and the UC’scertificate or a *.p12 certificate binary code, which contains the UC’s certificate and a private keyencrypted by a passphrase.

For example: cert_mgt import -jks uc.ks -pwd loglogic -cert uc-cert.pem -rootcert ca.pem


Universal Collector

Step 5: Configure the Forwarding ProcessIf the connection is authenticated or encrypted, the necessary cryptographic elements must be imported.

The three supported formats are:

• *.ks--A keystore in the JKS format containing the root CA, the private key and the associated UCcertificate.

Associated configuration elements are a keystore filename and a password for the keystore(mandatory)

• *.p12--A keystore in the PKCS#12 format, containing the private key and the associated UC certificateand the root CA (in *.pem format) as a separate file.

Associated configuration elements are a PKCS#12 (.p12) file, a password protected PKCS#12 file(mandatory) and a root CA file.

• *.pem--A private key (encrypted or not), a certificate to be used by UC in PEM format, a root CAcertificate in PEM format. Associated configuration elements are a private key file, a password if theprivate key is encrypted (mandatory), a UC certificate file, a root CA certificate file.

The Certificate Authority’s certificate allows to check the validity of the LMI or syslog server’s certificatetowards the UC.

The UC Valid certificate allows you to identify the UC from the LMI.

The Certificate Authority must be the one you previously used to validate the LMI or syslog servercertificate.


2. Click the New Connection button to open the Edition tab.

3. In the Description part of the screen, modify the name of the LMI or syslog server connection.

4. In the Security part of the screen, activate the following options:

Value Description

Authentication Activates the authenticated communication when the button is ON

Encryption Activates the encrypted communication when the button is ON

Certificate Displays the certificate imported in UC

Initialize Secured Connection Displays the screens to import the certificates

For *.ks

1. In the Secured Connection Initialization screen, select JKS and click Continue.


Universal Collector

2. In the Java Keystore section, click Import and select the UC JKS Certificate in *.jks format.

3. Enter the certificate password and click OK.

4. Click OK to close the window.The screen is automatically updated.

For *.p12

1. In the Secured Connection Initialization screen, select P12 and click Continue.

2. In the UC Certificate section, click Import and select the UC PKCS#12 Certificate in *.p12 format.

3. Enter the certificate password and click OK.

4. In the Root CA Certificate section, click Import and select the root CA certificate stored in *.p12format.


For *.pem

1. In the Secured Connection Initialization screen, select PEM and click Continue.

2. In the UC Certificate section, click Import and select the UC Certificate in *.pem format.

3. In the new small window, click Import Private Key and select the file in .pem format.

4. Enter the private key password and click OK.

5. In the Root CA Certificate section, click Import and select the root CA certificate stored in *.pemformat.


Configure the Forwarding Process

1. In the Forwarding part of the screen, modify the following values:

Forwarding

Address Enter the IPv4 address or host name of the LMI.

Port Select the LMI port or enter a port.


- 5515 for secured connection with LMI 5.0 or later (configurable in LMI)

- 5516 for connection with LMI 5.2 or later

Test connection Test the connection between UC and LMI.

Forward UC Internal Logs Define whether the UC internal logs are sent to the remote LMI byselecting ON.

Compress Messages If the connection is slow, you can configure the logs to be compressedfor a more rapid flow of data. Define whether the logs are compressed byselecting ON.

Advanced

Reconnection Enter the reconnection frequency to the LMI (in seconds)

Session timeout Enter the session timeout to LMI (in seconds)


Universal Collector

UC Binding interface If there are multiple network interfaces, enter the IP address that the UCuses when establishing the connection to LMI.

Default: 0.0.0.0

2. In the Message Buffering part of the screen, modify the following values:

Message Buffering


Scheduled Forwarding Define the period of time during which the logs are sent to the LMI (timewindow) by selecting ON.

Daily Start Define the beginning of the time window. If sendingWindow = true inthe above parameter, define the time (hour and minute) when the eventstarts to be sent (Default: 23:00)

Daily Stop Define the end of the time window. If you set sendingWindow = true inthe above parameter, define the time (hour and minute) when the eventstops to be sent (Default: 05:00).

3. Click OK to save and close the screen.The list of LMI connections is updated.

The configuration of UC has finished. When the certificate has expired, you must follow theprocedure from the beginning. You can use the same CSR you used if you have stored it earlier.


Universal Collector

Step 6: Enable Secure Connection

As for LMI, two certificates are needed:

• The Root CA, delivered by a certificate authority server. It will check the UC’s identity.• A certificate signing request or CSR. In order to generate the signed certificate, manual steps are

required unlike UC.

1. Using the LogLogic CLI, create a Certificate Signing Request:

system secureuldp create csr

This will generate a private key as well as the CSR.

The CSR is the value between the Begin Certificate and End Certificate lines.

2. If you have already created your CSR and just want to display it again, enter:system secureuldp show csr

3. Copy the CSR and sign the CSR. Once the CA signs the CSR, it will generate a signed certificate calledLMI.

4. Install this signed certificate back to the LMI Appliance by entering:system secureuldp install certificate

5. Paste the certificate in. Make sure to include the Begin Certificate and End Certificate lines whenpasting it in

6. Install the root CA certificate which will be the common certificate used for validation between the LMIand UC. To do so, enter:system secureuldp install rootCA

7. Paste it in the root CA certificate.

8. You may need to restart the application:mtask stop ; mtask start

9. Once you have created all the certificates, you must go to Administration > System Settings >General and check the Yes radio button associated with Enable Secure ULDP.

The communication between UC and LMI is now secured.

Managing the list of Forwardings | 71

Universal Collector

Managing the list of Forwardings

You can easily copy or delete Forwardings.

Label/Button Description

Name Label of the configuration

Address IPv4 address or host name of the server

Port Forwarding port

[ULDP only] UCLogs

Indicates whether the UC internal logs are sent to the remote LMI or not

[ULDP only] Comp. Indicates whether the logs are compressed or not

Auth. Communication authenticated or not

Encrypt Communication encrypted or not

Buffer (MB) Buffer size in megabytes (100 MB - default value, 50 GB - maximum value)

[ULDP only] Sched. Indicate if the messages are sending to the server during a specified time window

New Allows you to add new Forwardings to the list (Maximum 10)

Edit Allows you to edit Forwardings one by one

Copy Allows you to copy Forwardings to the list

Delete Allows you to delete Forwardings from the list


Universal Collector

Copying a ForwardingYou can copy a Forwarding one by one. The copied Forwardings keep the same configuration and thesame name with the _Copy suffix.

1. Select the Forwarding that you want to copy.

2. Click Copy.The new Forwarding is displayed in the Forwarding list. Double-click on the row to edit or modify theconfiguration.

By default, the Forwarding is linked with no Log Source.


Universal Collector

Deleting a ForwardingYou can delete Forwardings one by one.

1. Make sure that the Log sources linked to the Forwarding are removed or disabled.

2. Select a row from the list and click Delete. Click Yes to confirm.The list is automatically refreshed.


Universal Collector

Universal Collector

Chapter

4Monitoring UC Activities

A UCMon tool is also available to monitor the internal process of the UC.

This section provides instructions for quickly checking that UC is working properly, troubleshooting UC,Forwarding connection configuration, and monitoring the activities of the different log sources

Topics

• Starting UCMon Tool• Summary Screen• Status Screen• Metrics Screen• Trends Screen• RealTime Screen

Starting UCMon Tool

To start UCMon from UC Console

Open the UC Console and go to Manage Configuration > Monitor Active Configuration.

To start UCMon manually

Open the UC installation folder and launch the executable file located in the tools folder:

uc_monitor.exe (Windows) also available by clicking on the uc_monitor shortcut

uc_monitor (RHEL, SUSE or Solaris)

The UCMon is displayed.

Summary Screen

Label Description

Uptime Time when the UC has been started

Current Time Current date and time are automatically refreshed

Totals for the UC

Collected Total number of collected message for a given period of time

Summary Screen | 76

Universal Collector

Label Description

Between brackets, number of collected message per second

Filtered Total number of filtered message for a given period of time

Between brackets, number of filtered message per second

To Buffer Total number of forwarded message for a given period of time

Between brackets, number of forwarded message per second

UC Mem Current memory used / Total memory (Java Heap Size)

Config Current configuration name

Forwarding Connections and Log Sources

All Forwarding Conn. Forwarding connection status

• Active: the Forwarding connection works correctly• Idle: Forwarding connection is OK but the connection is NOT

established• Error: there is an error on the Forwarding connection• Off: indicates when the Forwarding connection is not used• Total: total number of enabled Forwarding connections

All Log Sources/Syslog/Windows Event Log/RT File/Remote File

Log Sources status

• Active: the Log Sources are answering correctly• Idle: Log Source not active at the moment• Error: there is an error on the Log Source• Off: indicates when a Log Source is inactive• Total: total number of Log Sources

Interactive menu

< C > Changes the time value of the “Totals for UC” metrics.

Each time you enter C, the value switches as follows:

• current value• 1 minute• 5 minutes• 15 minutes• 24 hours• time when the UCMon has been started

< M > Displays additional information

< 1 > Displays the Summary view

< 2 > Displays the Status view

< 3 > Displays the Metrics view

< 4 > Displays the Trend view

< 5 > Displays the Real Time view

Status Screen | 77

Universal Collector

Label Description

< Q > Quit the UCMon tool

Status Screen

To switch between Log Sources and Forwarding connection views, press L.

Status Screen | 78

Universal Collector

Log Source Status

Label Description



Log Source: Name of the Log Source

Status Status of the current Log Source:

• Active: the connection is OK• Err: the connection encountered an error• Idle: the connection never received a message from the source or

nothing at all for 24 hours• Off: a Log Source is inactive

Type Type of the Log Source: Win EL, RT File, Remote File or Syslog

Collection Connection parameters

• Win EL: Server IP or address• Syslog: protocol/binded port• RT File: Filename (no path)• Remote: File path

Forwarding Connection Current Forwarding connection associated with the current Log Source

Interactive menu

1-n/n Scrolls the tables into view

< N >ext/< P >revious Displays the next or previous view of the list of Log Sources

< E >rr first Sort Log Source status by Error (ERR) or alphabetical order

< V >erbose mode Display additional information

< L >og Source/Forwarding Switch between Forwarding connections and Log Sources tables







Metrics Screen | 79

Universal Collector

Forwarding Connection Status

Label Description



Forwarding Connection

Status Status of the current Log Source:

• Active: the connection is OK• Err: the connection encountered an error or spool may be full• Idle: no message transmitted from the source or nothing for 24 hours• Off: a Forwarding connection is not used

Address IP address and port of the remote Forwarding connection

S C A E Current Forwarding connection settings:

• S: Scheduled• C: Compression• A: Authentication• E: Encryption

Usage Spool load of the current Forwarding connection in %

Interactive menu


< N >ext/< P >revious Displays the next or previous view of the list of Forwarding connections

< E >rr first Sort Log Source status by Error (ERR) or alphabetical order

< V >erbose mode Display additional information








Metrics Screen

Metrics Screen | 80

Universal Collector


Metrics Screen | 81

Universal Collector

Log Source Metrics


Label Description




Format Format of the displayed values (messages or mps)

Period Period of time when the data are displayed (since uptime, 1min, 5 min,15 min, 24h)

Sort Sorting order of Log Source: By name/ In values (descending)

Forwarding Connection Define the current Forwarding connection with the Log Source

Collected Total number of collected message for a given period of time

Filtered Total number of filtered message for a given period of time

To Buffer Total number of forwarded message for a given period of time

Interactive menu



< F >ormat data Switch between message or messages per second.


< C >ycle period Switch of time period (current, 1mn, 5mn, 15mn, 24h, uptime)

< S >ort table Sort by collected values (descending) or by name







Trends Screen | 82

Universal Collector

Forwarding Connection Metrics

Label Description





Period Period of time when the data are displayed (since uptime, 1min, 5 min,15 min, 24h)

Sort Sorting order of Forwarding connection: By name/ In values (descending)

IN Input log rate

OUT Number of forwarded logs coming out from the spool

Usage Current Forwarding connection spool load

Interactive menu



< F> omat data Switch between message or messages per seconds


< C >ycle period Switch of time period (current, 1mn, 5mn, 15mn, 24h, uptime)

< S >ort table Sort by IN (descending) or by name







Trends Screen


Trends Screen | 83

Universal Collector

Log Source Trends

Label Description





Display Type of display. The possible values are:

• Collected• Filtered• Forwarded

Forwarding Conn. Name of the Forwarding connection

current, 1min, 5min, 1h, 24h,since uptime

Log rate over different time periods:

• n/a: value not available

Interactive menu



< F >ormat data Switch between message or messages per seconds


< D >isplay Displays the values for different probes:

Forwarding connection data type: IN or OUT

Log Source data type (Collected, Filtered, Forwarded)







RealTime Screen | 84

Universal Collector

Forwarding Connection Trends

Label Description



Forward Connection



• IN• OUT




Interactive menu


< N >ext/< P >revious Displays the next or previous Forwarding connection and Log Sources

< F >ormat data Switch between message or messages per seconds











RealTime Screen



Universal Collector

Log Sources RealTime

Label Description





• Collected• Filtered• Forwarded




Interactive menu














Universal Collector

Forwarding Connection RealTime

Label Description





• IN• OUT




Interactive menu













Universal Collector

Chapter

5Command Line Interface

The Command Line Interface (CLI) interacts with the local Universal Collector.

You can make a configuration active and reload the current configuration, check the current configuration,manage the security certificates, encrypt password or import several Log Sources in a row.

To start a Command Line Interface, open a shell into the following path:

Operating System CLI

Windows C:\Program Files\LogLogic\Universal Collector\tools\

RHEL, SUSE, Solaris /opt/LogLogic/Universal_Collector/tools/

The extension of the file to execute in order to run the commands differs on each UC supported OS:

Windows: uc_*.bat

RHEL, SUSE, Solaris: no extension

All the samples are given for RHEL, SUSE and Solaris environments. For Windows environment, use the samecommand with *.bat.

Topics

• cert_mgt: Manage the Security Certificates• uc_checkConf: Check the Current Configuration• uc_createLogSources: Import and Create Several Log Sources at a time• uc_decodePwd: Decode Passwords for Windows Files• uc_encryptPwd: Encrypt Passwords for Windows Files• uc_monitor: UCMon Tool• uc_reload: Reload Configuration• uc_saveActiveConfAs: Save an Active Configuration• uc_switchTo: Make Configuration Active

cert_mgt: Manage the Security Certificates

UC does not have to be started.

Goal Syntax Options

Request for *.pem cert_mgt request -pem <certfile> -csr <fileresult> -pwd

<password>

uc_checkConf: Check the Current Configuration | 88

Universal Collector

Goal Syntax Options

Request for *.ks cert_mgt request -jks <file.ks> -pwd <password> -csr

<fileresult.csr>

Request for *.p12 cert_mgt request -p12 <file.p12> -pwd <password> -csr

<fileresult.csr>

Import for *.ks cert_mgt import -jks <file.ks> -pwd <password> -cert

<certToImport> -rootcert <rootcertificate>

Import for *.p12 cert_mgt import -p12 <file.p12> -pwd <password> -cert

<certToimport>

Get help on thecertificates

cert_mgt

cert_mgt request

cert_mgt import

-h

Get information on thetool version

cert_mgt -v <nameofconf>

uc_checkConf: Check the Current Configuration

UC must be started.

Goal Syntax Options

Indicate validity ofthe configuration anddisplays potentialerrors and warnings

uc_checkConf -ucc <nameofconf>

Get help on the tool uc_checkConf -h

Indicate the port toconnect to the UC

uc_checkConf -ucc <nameofconf> -p <portnumber>

Get information onthe tool version

uc_checkConf -v

uc_createLogSources: Import and Create Several LogSources at a time


uc_decodePwd: Decode Passwords for Windows Files | 89

Universal Collector

Goal Syntax Options

Indicate the typeof Log Sources toimport (WindowsEvent Log, syslog,file, remotefile)

uc_createLogSources -t <windows,

syslog, file,

remotefile>

Import a CSV filewith Log Sourceinformation to createa Log Source

uc_createLogSources -in <pathname>

Indicate the *.uccfile where to exportthe Log Sourceinformation

uc_createLogSources -out <pathname>

Force the commandwithout anyconfirmation

uc_createLogSources -f

uc_decodePwd: Decode Passwords for Windows Files


Goal Syntax Options

Allows decodingpassword

/opt/LogLogic/UniversalCollector/tools/uc_decodePwd

<passwordTodecode>

uc_encryptPwd: Encrypt Passwords for Windows Files


Goal Syntax Options

Allows encodingpassword

/opt/LogLogic/UniversalCollector/tools/uc_encryptPwd

<passwordToencrypt>

uc_monitor: UCMon Tool

uc_reload: Reload Configuration | 90

Universal Collector


Goal Syntax Options

Indicates the UCport to which theUCMon listens to (ifnot default port)

/opt/LogLogic/UniversalCollector/tools/uc_monitor -p

<portnumber>

-p <portnumber>

uc_reload: Reload Configuration

UC must be started.

This command is used to update the active configuration without stopping the whole process.

To update the current configuration, the command is:

For Windows:uc_reload.bat

For RHEL, SUSE, Solaris:uc_reload

Example 1: You want to update the active configuration ‘conf1’.

Enter the command to apply a new configuration to the UC via the CLI located in <INSTALL_DIR>/tools.

\uc_reload.bat

The active configuration is updated.

Example 2: You want to check the impacted process during an update of the configuration.

Enter the following command:

uc_reload.bat -dryrun -vb

Goal Syntax Options

Reload the currentconfiguration to applychanges

uc_reload

There is no need to enter the name of the configuration as it is the current configuration, which isautomatically updated.

uc_saveActiveConfAs: Save an Active Configuration


uc_switchTo: Make Configuration Active | 91

Universal Collector

Goal Syntax Options

Save a configurationcurrently in use

uc_saveActiveConfAs <pathname\confname.ucc>

Force to savea configurationcurrently in use evenif it already exists

uc_saveActiveConfAs <pathname\confname.ucc> -f

uc_switchTo: Make Configuration Active

UC must be started.

Goal Syntax Options

Activate UC Configuration uc_switchTo -ucc <nameofconf>

Simulate the change of theactive UC configuration.Displays possible errorsand warnings in the storedconfiguration and changesbetween active and storedconfigurations.

uc_switchTo -ucc <nameofconf> -dryrun

Get help on the Switchcommand

uc_switchTo -h

Indicate the port to connect tothe UC

uc_switchTo -ucc <nameofconf> -p <portnumber>

Get information on the Switchversion

uc_switchTo -v

Activate UC Configuration anddisplay verbose information

uc_switchTo -ucc <nameofconf> -vb

Switching from One Configuration to Another

It is possible to switch from one configuration to another one.

To apply a new configuration, the command is:

uc_switchTo.bat -ucc {myconf}(under Windows)

uc_switchTo -ucc {myconf} (under RHEL, SUSE, Solaris)

In case of an error, the configuration switch is interrupted and the configuration error is logged in the uc.logfile.

Example: You want to switch from the current configuration ‘conf1’ to ‘conf2’.

Enter the command to apply a new configuration to the UC via the CLI located in <INSTALL_DIR>/tools.

\uc_switchTo.bat -ucc c:\tmp\conf2

uc_switchTo: Make Configuration Active | 92

Universal Collector

The current configuration is now ‘conf2’.

Checking the Impacted Processes

It is possible to check which log sources and Forwarding connections are impacted by the newconfiguration - without having to apply it.

To check the impact on the processes:

-dryrun gives information on the switch or the update of configurations

-dryrun -vb gives detailed information on the switch or the update of configurations

Example: You want to check the impacted process during a switch of configurations.

Enter the following command:

uc_switchTo.bat -ucc {uc.conf.file}.ucc -dryrun -vb

You can obtain something like this:

3 configuration files checked

1 Log Source config updated

1 SYSLOG Log Source config updated

2 Forwarding connection updated (1 created, 1 removed)

1 LS Config Updated

============================================================

syslog.1 UPDATE

2 Forwarding Config Updated

============================================================

MyCuteLmi2 REMOVE

MyCuteLmi CREATE

WARNING data may not have been collected during the switch configuration operation, the log sources

[syslog.1] may have been impacted

WARNING data contained in Forwarding connection spool of [MyCuteLmi2] may have been lost if remote

Forwarding connection was not available

SUCCESS-[conf3] DryRun mode : No change has been applied to the running configuration

Limitations

During a switch process, some limitations may occur.

• First case--If you remove or update a Syslog Log Source, you may stop the flow and lose some data.

• Second case--If you switch from a Forwarding connection to another one for a given Syslog LogSource, you may lose a few events. This behavior is rare though.

• Third case--If you remove a Forwarding connection or modify the values of the buffer size while theconnection to the Forwarding connection is not available (for example, network failure), the Forwardingconnection buffer will try to empty itself by sending the remaining data to the Forwarding connection.This will cause the loss of the buffer content during the time-out.

Universal Collector

Appendix

ASample Configuration Files

In the installation directory, the folder <config-samples> contains the templates you can copy to create a completeconfiguration manually without using UC Console.

• sample-commented.ucc contains documented XML files.• sample-lite.ucc contains XML files with mandatory tags only without documentation.• sample.ucc contains XML files with all the tags without documentation.

When you unzip one of them, you obtain:

• uc.xml file: allows the configuration of the UC’s general information.• log-sources sub-folder: contains documented templates to define a log source, it is what you can find under

the Collection tab in the GUI.• uldp sub-folder: contains documented templates to define the Forwarding connections. It is what you can find

under the Forwarding tab or when editing a Forwarding Connection in the GUI.

Topics

• [UC Configuration] uc.xml• [LMI Connection]uldp-sampleCommented.uldp.xml• [LMI Connection] uldp-sampleCommentedAuthJks.uldp.xml• [LMI Connection] uldp-sampleCommentedAuthPem.uldp.xml• [LMI Connection] uldp-sampleCommentedAuthPks12.uldp.xml• [Log Sources] file-sampleCommented.ls.xml• [Log Sources] syslog-sampleCommented.ls.xml• [Log Sources] wmi-sampleCommented.ls.xml

Sample Configuration Files | 94

Universal Collector

[UC Configuration] uc.xml

You must unzip sample.ucc to display the uc.xml file, which contains the information you can find under theGeneral Settings tab in the GUI.

 <uc schemaVersion="2.0">



<configurationName>sampleCommented</configurationName>



<domainName>sampleDomainName</domainName>



<ucCommunicationPort>1099</ucCommunicationPort>



<syslogCollection>



<socketBufferSize>1024</socketBufferSize>



<udpMaxPacketSize>8</udpMaxPacketSize>

</syslogCollection>

</uc>

Universal Collector

[LMI Connection]uldp-sampleCommented.uldp.xml



<uldpConnection schemaVersion="2.0">



<name>Full_ULDP_File</name>



<revision>



<version>12</version>



<author>admin</author>



<creationDate>2011-04-22T01:00:00-05:00</creationDate>



<lastModifiedBy>admin</lastModifiedBy>



<lastModifiedDate>2011-04-22T01:00:00-05:00</lastModifiedDate>

</revision>



<address>192.198.12.16</address>



<port>5514</port>



<compression>true</compression>



<sendingWindow>true</sendingWindow>

Universal Collector



<sendingWindowStart>22:00</sendingWindowStart>



<sendingWindowStop>05:00</sendingWindowStop>



<authentication>false</authentication>



<encryption>false</encryption>



<uldpForwarder>



<ucBindingIp>0.0.0.0</ucBindingIp>



<spoolerSize>100</spoolerSize>



<reconnectionFrequency>60</reconnectionFrequency>



<sessionTimeout>600</sessionTimeout>



<internalUcLogs>false</internalUcLogs>

</uldpForwarder>

</uldpConnection>

Universal Collector

[LMI Connection] uldp-sampleCommentedAuthJks.uldp.xml







<revision>












</revision>





- 5515 for secured connection LMI 5.0 or later


<port>5515</port>

Universal Collector








<authentication>true</authentication>





<certificate>

<jks>



<jksFile>sample.jks</jksFile>



<password>LSKS9bw/t01FqNd4P3l3pgeOy/N/qqqlEzG+QC/kfDq0LVXTPVgziQ==</password>


-->


</jks>

</certificate>

</uldpConnection>

Universal Collector

[LMI Connection] uldp-sampleCommentedAuthPem.uldp.xml







-->





<revision>












</revision>







<port>5515</port>

Universal Collector













<certificate>


-->


<pem>



<pemPrivKeyFile>pemPrivKeyFile</pemPrivKeyFile>






<pemCertFile>pemCertFile</pemCertFile>



<pemRootCertFile>pemRootCertFile</pemRootCertFile>

</pem>

</certificate>

</uldpConnection>

Universal Collector

[LMI Connection] uldp-sampleCommentedAuthPks12.uldp.xml







-->





<revision>












</revision>







<port>5515</port>

Universal Collector













<certificate>

<pkcs12>



<p12CertFile>p12CertFile</p12CertFile>






<pemRootCertFile>pemRootCertFile</pemRootCertFile>


-->


</pkcs12>

</certificate>

</uldpConnection>

This file is located in <InstallFolder>\config-samples\.

You must unzip sample.ucc and open the log-sources folder.

Universal Collector

[Log Sources] file-sampleCommented.ls.xml





<logsource type="file" schemaVersion="2.0">

<general>



<active>true</active>



<name>ls-file-template</name>



<description>Comment of the ls-file-template</description>



<revision>
















</revision>

</general>



<forwarding>



<uldp>

Universal Collector

<connectionId>uldp-sampleCommented</connectionId>



<timeInUtc>false</timeInUtc>

</uldp>

</forwarding>



<collection>



<maxLineLength>65000</maxLineLength>



<charsetName></charsetName>



<fileName>



<absolutePath>c:\temp\logfile.log</absolutePath>



<useDateRolling>false</useDateRolling>



<dateFormat>yyyyMMdd</dateFormat>



<useIdRolling>false</useIdRolling>



<nbDigit>2</nbDigit>

</fileName>

</collection>



<processing>



<multiLine>

Universal Collector

<active>false</active>



<lineCombinerId>jboss</lineCombinerId>



<userDefinedRegExp></userDefinedRegExp>



<keepHeadlessLog>false</keepHeadlessLog>



<lineTimeout>3000</lineTimeout>

</multiLine>



<hostname>customHostname.com</hostname>



<appName>customApplicationName</appName>

</processing>



<filter>



<messageFilter>packet accepted</messageFilter>



<matchAcceptedMessage>false</matchAcceptedMessage>

</filter>



<tags>



<tag>sample</tag>

<tag>commented</tag>

</tags>

</logsource>

Universal Collector

[Log Sources] syslog-sampleCommented.ls.xml




<logsource type="syslog" schemaVersion="2.0">

<general>





<name>ls-syslog-template</name>



<description>Comment of the ls-syslog-template</description>



<revision>













</revision>

</general>


<forwarding>


-->

<uldp>


<connectionId>uldp-sample</connectionId>

Universal Collector




</uldp>

</forwarding>


<collection>



<ip>0.0.0.0</ip>



<port>514</port>



<protocol>udp</protocol>

</collection>



<filter>



<severity>6</severity>



<facilities>0-23</facilities>



<sourceIp>.*</sourceIp>

</filter>


<tags>


<tag>sample</tag>


</tags>

</logsource>

Universal Collector

[Log Sources] wmi-sampleCommented.ls.xml




<logsource type="wmi" schemaVersion="2.0">

<general>





<name>ls-win-template</name>



<description>Comment of the ls-win-template</description>



<revision>
















</revision>

</general>


<forwarding>


-->

<uldp>


<connectionId>uldp-sampleCommented</connectionId>

Universal Collector




</uldp>

</forwarding>


<collection>



<domain>domain.company</domain>






<login>jdoe</login>






<pollingPeriod>10</pollingPeriod>

</collection>



<filter>



<includeJournal>only</includeJournal>



<journalList>

<journal>Security</journal>

<journal>Application</journal>

</journalList>



<eventIdFilter>.*</eventIdFilter>



<sourceFilter>.*</sourceFilter>

<!-- Enter the filter operator for the <eventIdFilter> and <sourceFilter> tags, It can be either:


Universal Collector

- both filters: and (default value)

- only one: or

-->

<filterOperator>and</filterOperator>

</filter>


<tags>


<tag>sample</tag>


</tags>

</logsource>

Universal Collector

Appendix

BRegular Expressions

Regular expressions provide a concise and flexible means for “matching” (specifying and recognizing) stringsof text, such as particular characters, words, or patterns of characters. They are used when you configure LogSources.

Construct Matches

Characters

x The character x

\ \ The backslash character

\0n The character with octal value 0n (0 <= n <= 7)

\0nn The character with octal value 0nn (0 <= n <= 7)

\0mnn The character with octal value 0mnn (0 <= m <= 3, 0 <= n <= 7)

\xhh The character with hexadecimal value 0xhh

\uhhhh The character with hexadecimal value 0xhhhh

\t The tab character ('\u0009')

\n The newline (line feed) character ('\u000A')

\r The carriage-return character ('\u000D')

\f The form-feed character ('\u000C')

\a The alert (bell) character ('\u0007')

\e The escape character ('\u001B')

\cx The control character corresponding to x

Character classes

[abc] a, b, or c (simple class)

[^abc] Any character except a, b, or c (negation)

[a-zA-Z] a through z or A through Z, inclusive (range)

[a-d[m-p]] a through d, or m through p: [a-dm-p] (union)

[a-z&&[def]] d, e, or f (intersection)

[a-z&&[^bc]] a through z, except for b and c: [ad-z] (subtraction)

[a-z&&[^m-p]] a through z, and not m through p: [a-lq-z] (subtraction)

Regular Expressions | 112

Universal Collector

Construct Matches

Predefined character classes

. Any character (may or may not match line terminators)

\d A digit: [0-9]

\D A non-digit: [^0-9]

\s A whitespace character: [\t\n\x0B\f\r]

\S A non-whitespace character: [^\s]

\w A word character: [a-zA-Z_0-9]

\W A non-word character: [^\w]

POSIX character classes (US-ASCII only)

\p{Lower} A lower-case alphabetic character: [a-z]

\p{Upper} An upper-case alphabetic character:[A-Z]

\p{ASCII} All ASCII:[\x00-\x7F]

\p{Alpha} An alphabetic character: [\p{Lower}\p{Upper}]

\p{Digit} A decimal digit: [0-9]

\p{Alnum} An alphanumeric character: [\p{Alpha}\p{Digit}]

\p{Punct} Punctuation: One of !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~

\p{Graph} A visible character: [\p{Alnum}\p{Punct}]

\p{Print} A printable character: [\p{Graph}]

\p{Blank} A space or a tab: [\t]

\p{Cntrl} A control character: [\x00-\x1F\x7F]

\p{XDigit} A hexadecimal digit: [0-9a-fA-F]

\p{Space} A whitespace character: [\t\n\x0B\f\r]

Classes for Unicode blocks and categories

\p{InGreek} A character in the Greek block (simple block)

\p{Lu} An uppercase letter (simple category)

\p{Sc} A currency symbol

\P{InGreek} Any character except one in the Greek block (negation)

[\p{L}&&[^\p{Lu}]] Any letter except an uppercase letter (subtraction)

Boundary matchers

^ The beginning of a line

$ The end of a line

\b A word boundary


Universal Collector

Construct Matches

\B A non-word boundary

\A The beginning of the input

\G The end of the previous match

\Z The end of the input except for the final terminator, if any

\z The end of the input

Greedy quantifiers

X? X, once or not at all

X* X, zero or more times

X+ X, one or more times

X{n} X, exactly n times

X{n,} X, at least n times

X{n,m} X, at least n but not more than m times

Reluctant quantifiers

X?? X, once or not at all

X*? X, zero or more times

X+? X, one or more times

X{n}? X, exactly n times

X{n,}? X, at least n times

X{n,m}? X, at least n but not more than m times

Possessive quantifiers

X?+ X, once or not at all

X*+ X, zero or more times

X++ X, one or more times

X{n}+ X, exactly n times

X{n,}+ X, at least n times

X{n,m}+ X, at least n but not more than m times

Logical operators

XY X followed by Y

X|Y Either X or Y

(X) X, as a capturing group

Back references

\n Whatever the nth capturing group matched


Universal Collector

Construct Matches

Quotation

\ Nothing, but quotes the subsequent character

\Q Nothing, but quotes all characters until \E

\E Nothing, but ends a quote started by \Q

Special constructs (non-capturing)

(?:X) X, as a non-capturing group

(?idmsux-idmsux) Nothing, but turns match flags on - off

(?idmsux-idmsux:X) X, as a non-capturing group with the given flags on - off

(?=X) X, via zero-width positive look ahead

(?!X) X, via zero-width negative look ahead

(?<=X) X, via zero-width positive look behind

(?<!X) X, via zero-width negative look behind

(?>X) X, as an independent, non-capturing group

tibco loglogic universal collector user guide · tibco loglogic® universal collector (uc) is a...

Documents