Normalizing Data for use in Forensics Analysis

Major Activities

XML Parser for the Application Log data provider

Metadata map

Forensics template

Modified Forensics.xml file to make custom forensics template available in the Control Center Console

Rule to archive data from the application log

Custom XML Parser

Metadata

Used to tag events with information to store the data in an IDMEF compliant manner in the database

− IDMEF – Intrusion Detection Message Exchange Formathttp://www.rfc-editor.org/rfc/rfc4765.txt

Used to correlate different event properties (i.e. Source IP from two different events)

Maps are built in an XML file and imported into the database

Mapping is stored in the OnePoint database− MetadataFieldDefinition− MetadataLocalization− MetadataMap− MetadataMapMain

Field Map Used to uniquely identify fields to be used in mapping Each field that can be used in a metadata map is defined here Beginning in SM 6, the field map can no longer be modified

− Any prior modifications will be removed.

Datatypes are restricted in the field map Example mapping of standard IDMEF field:

<field id="501" IDMEF="analyzer.model" datatype="string"><Locale id="1033">

<DisplayName>Device Model</DisplayName>

</Locale></field>

− Example mapping for an additional data field:<field id="915" IDMEF="Signature Name" datatype="string">

<Locale id="1033"><DisplayName>Signature

Name</DisplayName></Locale>

</field>

The Hierarchy of Maps

Metadata Indexes must be unique at each level

As an event flows down the system, additional properties are mapped

The most specific mapping definition is the one that will be applied.

Metadata Hierarchy (Cont…)

Catchall – used only by NetIQ to ensure that specific fields are mapped so that they are available to the system for reporting purposes

Event Source - must be unique throughout the entire system

Event ID - Metadata Index should start at 1 and must not be repeated per Event Source

Dll Specific - last level and it follows the same rules as Event ID

Constructing the Map – EventIDMap

<EventIDMap name=“TippingPoint” id=“0” version=“5.50.0.158”>

</EventIDMap>

Name is an arbitrary name for the map (and cannot be renamed)

Can leave the id = 0 since SM will automatically assign this value when you import the map

Version is used in the import process− When incrementing the version number do not make major

leaps in the versioning. Use the lowest level version segment to increment it.

Constructing the Map – Source<Source id=“TippingPoint” MetadataIndex=“1002”>

<Params>

Define Parameters here that are consistent throughout the event source

</Params>

</Source>Source id MUST match the value you put for the event source in the XML parser in order for SM to determine that this is the appropriate map for that event

MetadataIndex number must be unique (for custom maps, use ID numbers 1000-9999).

Don’t use ID 1001!

Constructing the Map – Event<Event id=“200” MetadataIndex=“1”>

<Params>

Define Parameters specifically for this event id.

</Params>

</Event>Event IDs should align to what has been defined in the parser.

MetadataIndex always start at “1” at the Event level and increase for each new event defined

Constructing the Map – DLL Specific<DLLSpecific dllname="MSAuditE.dll" start="5.0.2149.1"

end="6.0.0.0" MetadataIndex="1">

<Params>

Define Parameters here that are consistent throughout the particular message dll.

</Params>

</DLLSpecific>DllSpecific is used for events coming from a Windows NT event log. Requires the raw name of the msg dll. MSAuditE.dll is the Security Log. SM_Msgs.dll is the Security Manager dll. The start and stop are the version ranges of the dll that will have this map applied.

MetadataIndex always start at “1” at the Event level and increase for each new event defined

Constructing the Map – Defining Parameters

The Params section is the where the mapping to IDMEF actually occurs.

<Params><param field=“source.node.address.address” source=“6”/><param field=“target.node.address.address” source=“8”/><param field="Logon Type" source="4" additionaldata="true" />

</Params>

The numbers refer to parameters - “6” = “Parameter 6”

Parameters that do not have a specific IDMEF Field have to be identified as being additional data by setting additionaldata = “true”.

Supported Lookups & Operators

Lookups− Lookup=“SID” - resolves the SID contained in the specified

parameter <param field="target.user.userid.name" source="1" lookup="SID"/>

− Lookup=“IP” - reverse lookups the host name in the specified parameter <param field="source.node.Address.address" source="6" lookup="IP"/>

NOTE: Using IP Address lookups can be expensive. If the name resolution fails, it has to time out before the map can be complete

Operators− string literals: <param field=“X" source=“’foo’"/>− concatenate: <param field="source.user.userid.name" source="2 +

'\' + 1"/>

− conditional:<param field="classification.name" source="if(6 = ‘foo') Then 5 else 3"/>

Dynamic Event Properties Dynamic event properties are pieces of each event that are typically

defined in the workflow.

$GUID $ProviderInstance $EventNumber $EventType $EventTime $Category $ProviderName $Message $UserName $UserDomainName $Domain $Computer $AgentComputer $AgentDomain $SourceName $MessageDLL $DllVersion $Description $EventImportance $AnalyzerModel

Example of a Completed Map<EventIDMap name="Security Event Log" id="0" version="5.10.0.154">

<Source id="Security" MetadataIndex="0">

<Params>

<param field="message" source="$Message" additionaldata="true" />

<param field="analyzer.model" source="'Windows'" />

</Params>

(...)

<Event id="642" MetadataIndex="43">

<Params>

<param field="CLS" source="'PENTASAFE:3.2.2.55'" />

<param field="source.node.name" source="$Computer" />

<param field="source.node.Address.address" source="$Computer" lookup="IP" />

</Params>

<DLLSpecific dllname="MSAuditE.dll" start="5.0.2149.1" end="6.0.0.0" MetadataIndex="1">

<Params> <!-- Windows 2000 and later  -->

<param field="target.user.userid.name" source="2" />

<param field="source.user.userid.name" source="5" />

<param field="Account Change" source="1" additionaldata="true" />

<param field="target_account_id" source="4" lookup="SID" additionaldata="true" />

<param field="target_object_domain" source="3" lookup="SID" additionaldata="true" />

</Params>

</DLLSpecific>

<DLLSpecific dllname="MSAuditE.dll" start="4.0.1371.1" end="5.0.0.0" MetadataIndex="2">

<Params> <!-- Windows NT 4.0  -->

<param field="target.user.userid.name" source="1" />

<param field="target_account_id" source="3" lookup="SID" additionaldata="true" />

<param field="source.user.userid.name" source="4" lookup="SID" />

</Params>

</DLLSpecific>

</Event>

(...)

</Source>

</EventIDMap>

Parameters of Interest

Analyzer.model− This is an arbitrary name. Typically this will be the data provider

name.

Analyzer.node.name− This should be set to equal the parameter value indicating the device

that generated the event.

Classification.name− This should be set to equal the parameter value of the Event ID

number.

Any additional IDMEF field that is a part of the Reporting Cube

MetadataImporter.exe

This tool is contained in C:\Program Files\NetIQ Security Manager\Onepoint.

Be careful….this tool has limited error checking capabilities – just because it says an import completed successfully does not necessarily mean that it did!

Forensics Template

Allows you to generate platform specific forensic reports

Think about what fields you want to include in the report and associate them with the appropriate IDMEF data.

Must be copied to all UI machines in the following directory: − <InstallLocation>\NetIQ Security Manager\Onepoint\VSOC\

Config directory

Forensics Template

Notice the syntax for “nonstandard” IDMEF metadata fields. Just like in the metadata map, you need to have a special syntax associated:

The syntax is:

<Field name=“Forensic label” idmef=“ADDITIONALDATA.DATA” add_meaning=“nonstandardIDEMEFfieldname” />

Forensic label = whatever you wanted the display name to be in the template

nonstandardIDMEFfieldname = whatever you called the nonstandard field in the metadata map.

Forensics.xml

Stored in the C:\Program Files\NetIQ Security Manager\Onepoint\VSOC\Config directory

Makes the custom forensics template you already created available in the dropdown menu

Add a row to the file Containing the name of the report type, the PRG GUID, and the name of the template XML file.

Note for 5.6 – the network node MUST be a member of an SM Computer Group in order for report to populate properly.

Forensics.xml

Row Name = Arbitrary display name for the forensic template

PRGID = GUID of the PRG containing archive rules for the provider. Note: SM will list the computer groups associated with this GUID when defining a forensic query. The analyzer nodes that are returned by the query MUST be contained in one of these computer groups, or the data will not display in SM 5.6 (this is less of an issue in 5.0-5.5 since there is no security filtering on archival).

AnalyzerModel = Whatever was defined as the analyzer model in the metadata map

FILENAME = name of the XML file used for the Forensic template

Adding Custom Computer Groups to the Cross Platform Template

The Cross Platform forensic template has a PRGID of “00000000-0000-0000-0000-000000000000” allowing it to return all of the computer groups associated with a “NetIQ” module.   

− The forensics wizard queries the Modules tables and uses the idProcessRule to in turn look up all of the computer groups tied to a module. 

The correct way to address this is to insert a record into the Module table that builds the association to your custom PRG.  Here is a SQL query that will handle this insertion for you:

DECLARE @RuleGUID uniqueidentifierDECLARE @ModuleName nvarcharDECLARE @Version nvarcharSET  @RuleGUID = 'F1D5A4A5-C05D-406E-BC61-7BB4658FAFAA'  --Insert the idProcessRule of your PRG here.SET @ModuleName = <Some Name>' --Insert a name for the module here.SET @Version = '5.60.001' --Insert a version number here. ------------Do not edit anything below this line------------ Insert INTO Module (idModule, Name, Description, idProcessRule, FeatureCode, Reserve, Lastmodified, Version, idNQMModule)VALUES  (NEWID(), @ModuleName, '', @RuleGUID, '', '', GETUTCDATE(), @Version, NULL)

Make sure you have a description in the Computer Group or you will create an unhandled exception in the Forensic Wizard. 

Adding endpoints to CG outside of Config Wizard To force inclusion in the computer group outside of

a Configuration Wizard, run this stored procedure against the Onepoint database via SQL Query Analyzer:exec dbo.ComputerInsertNetworkNode 'domainname',

'computername', 'computergroupname' Or (if endpoint does not have a domain membership)exec dbo.ComputerInsertNetworkNode NULL, 'computername',

'computergroupname‘

For example, to add the device “10.1.1.14” to the Snort Intrusion Detection computer group, you would run:exec dbo.ComputerInsertNetworkNode NULL, ’10.1.1.14', 'Snort

Intrusion Detection'

Best Practices

Before you even start on the XML parser, try to think about what IDMEF fields you want to report on. Keep notes on what IDMEF parameters these IDMEF fields will map to when constructing your parser. *Remember the dimensions used in the SMReporting OLAP Cube.

When making your parser, try to make sure that a given metadata value will always map to the same event parameter. This makes the metadata map a lot simpler.

Note that importing metadata maps will NOT overwrite what is already in the Onepoint database unless the version is a later version

Best Practices - cont

You can test any text based log with a syslog simulator

NEVER modify the field map

Datatypes are restricted in the field map

Watch out if you want to use lookups in the metadata map

Be constantly aware of the dimensions available in the cube. These are the only fields that will be summarized.

And did I mention “Be constantly aware of the dimensions available in the cube.”

Report DimensionsNetwork Nodes Has a user hierarchy for Model and Name. It also has Domain and the GUID as attributes of it.

UTC Date Has a user hierarchy with 4 levels for Year, Quarter, Month and Day

Local Date This is computed using the UTC date and the offset in minutes. Has a user hierarchy with 4 levels for Year, Quarter, Month and Day

Source Address Will not be allowed to be part of any aggregate.

Target Address Will not be allowed to be part of any aggregate.

Source User  

Target User  

Severity 2 different severities (from a field map point of view) are combined in to one dimension. The field map fields for this one are ‘Severity’ and ‘Syslog severity’.

Event Classification User hierarchy of 3 columns namely Event Category, Classification and the ID itself.

Event Source If a given model can have multiple event source, this dimension will capture it. E.g., for windows module, the values could be security/application/system.

Source Port  

Target Port  

Source Protocol  

Target Protocol  

Source Interface  

Target Service Name  

Source Node Name  

Target Node Name  

Action Consolidated dimension that will get values from multiple module specific fields. E.g. firewall actions, operation, command, etc.

Rule Every firewall has its own way of tagging the rule. So this is a combined dimension which contains different column (from a field map point of view) that represent the same value. E.g rule id, rule name, rule number.

Timing Type Currently supports 4 possible values are Business hours, Non Business hours, Lunch Hours and Week ends. The timings are driven by the Trend analysis time properties in Global settings\Log Manager\Adanced Tab.

Object This dimension will contain File/registry/AD account names. The type will define what object it is.