new and noteworthy in sqrrl enterprise - version 2.9 · new and noteworthy in sqrrl enterprise 5...

New and Noteworthy inSqrrl Enterprise

Version 2.9.4

New and Noteworthy in Sqrrl Enterprise: Version 2.9.4Copyright © 2013-2019 Sqrrl Data, Inc.

Proprietary Information. All Rights Reserved.

Sqrrl is a trademark of Sqrrl Data, Inc.

New and Noteworthy in Sqrrl Enterprise 3

Table of Contents

New in Sqrrl Enterprise 2.9 ........................................................................... 5Enhancements to Sqrrl's risk trigger functionality ............................................ 5

New risk trigger list provides a dedicated location for managing risk triggers ..... 5New risk trigger profile provides a detailed summary of a risk trigger'sresults .......................................................................................... 6New options to include or exclude risk from a risk trigger ........................... 7New risk triggers to detect spikes in activity during each hour of the day ......... 7New risk trigger to detect entity instances with a spike in connectedrelationship instances ....................................................................... 8

New built-in relationships track when user agents are used to request IP addressesand DNS domains ................................................................................... 8Enhancements to the Sqrrl query language and the Sqrrl shell ............................ 9

New option to change the time interval when querying time series ................ 9

New in Sqrrl Enterprise 2.8 ......................................................................... 10New source abstraction layer to more easily map and query source data ............... 10New query-based risk triggers allow for more customized detection of interestingactivity .............................................................................................. 14Redesigned risk dashboard provides high-level overview of both detector and risktrigger results ..................................................................................... 16New risk timeline on entity instance profile traces the history of an entityinstance's risk ...................................................................................... 17New Sqrrl Flume receivers and Flume flows to load data directly from Netflow andIPFIX feeds, support Sqrrl integrations ........................................................ 19New support for data from threat intelligence feeds ...................................... 20New option to automatically increase a source shard count .............................. 21Other enhancements to the Sqrrl web application ......................................... 22

New exploration graph actions menu provides easier access to graph options ... 22Streamlined graph expansion options for the CounterOps model .................. 23Moved tags to a more prominent location on detection result and entityinstance profiles ............................................................................ 24

Enhancements to the Sqrrl shell ............................................................... 24Relationship list now includes the origin and destination entities ................. 24Including the job type in lists of job configurations and job instances ............ 25New option to view query function help from the Sqrrl shell ....................... 26Pagination control extended to all Sqrrl shell and administrative shellcommands ................................................................................... 26New option to edit configuration objects directly from the Sqrrl shellcommand line ............................................................................... 27New automatic detection of the UUID to update for the update command ...... 27New name/template ID macros for references to other configuration items ..... 28New compact display format for time series values ................................. 29

New in Sqrrl Enterprise 2.7 ......................................................................... 30

New and Noteworthy in Sqrrl Enterprise


DNS-related source connector and detectors added to the Sqrrl hunting tool set ...... 30New source connector to store Microsoft Server DNS Debug entries ............... 30New detector to find instances of DNS tunneling ..................................... 30New detector to find instances of domain generation algorithm activity ......... 31Built-in whitelist of common registered domains reduces false positives forDNS-related detectors ..................................................................... 32New built-in report elements for DNS data ............................................ 33

Search and query enhancements ............................................................... 33New combined search field speeds up searches for tags and instance IDs ........ 33New expanded query panel for the Explore context ................................. 34

Source connector, load jobs for SIEM integration now built into Sqrrl ................... 35New ARRAY data type for source fields ...................................................... 36


New in Sqrrl Enterprise 2.9

Enhancements to Sqrrl's risk triggerfunctionality

New risk trigger list provides a dedicated location formanaging risk triggers

Previously, risk triggers were in the exploration list along with the saved queries andinvestigations.

Risk triggers are now listed on their own risk trigger list page, which you display using a newoption on the main Sqrrl menu.

The new risk triggers list includes the option to create a new risk trigger, which is no longeron the exploration query panel. The list is separated into triggers that you created, and cantherefore edit and delete, and triggers created by other users, which you can only enable/disable.



New risk trigger profile provides a detailed summary of arisk trigger's results

For all risk triggers, the new risk trigger profile provides a detailed summary of:

• A summary of the risk trigger configuration. In the header, an orange gear icon indicatesthat the risk trigger is developmental. See New options to include or exclude risk from arisk trigger.

• The results from the last 180 days

You can navigate to the risk trigger profile from:

• The risk trigger list

• The risk trigger bar chart on the risk dashboard

The risk trigger profile includes:

• A synopsis of the risk trigger configuration

• A summary of the matches over the last 180 days

• A list of matching entity or relationship instances from a selected time window (the lastday, 7 days, 30 days, 90 days, or 180 days)

From the risk trigger profile, you can:

• Display details for a matching entity or relationship instance

• Navigate to the profile for an entity instance in the match list

• Navigate to an exploration graph that shows the entity and relationship instances fromselected matches



New options to include or exclude risk from a risktrigger

When you are working with a new risk trigger, and are not yet sure of the results, you maywant to see what entity instance risk scores look like both with and without that risk triggerincluded.

For 2.9, Sqrrl introduces a new developmental mode for risk triggers, configured on the risktrigger dialog.

When a risk trigger is developmental, then by default the risk from that risk trigger is notincluded when calculating risk scores.

A new option in the Sqrrl dropdown menu allows each user to indicate whether they want toinclude the risk from developmental risk triggers.

When a user opts to include risk from developmental risk triggers, the Sqrrl header changescolor and displays a a white, gear-shaped icon.

New risk triggers to detect spikes in activity during eachhour of the day

The existing rate spike risk triggers look for overall spikes in activity compared to a historicalbaseline.



Sqrrl 2.9 introduces new risk triggers for entity features and entity connections that look forspikes in activity during each hour of the day compared to the historical baseline for eachhour.

For example, an IP address usually has around 3 login attempts between 1 and 2 PM. But onthe day being analyzed, there are 50 login attempts during that hour.

New risk trigger to detect entity instances with a spike inconnected relationship instances

Sqrrl 2.0 introduces a new risk trigger, the degree spike risk trigger, that allows you to lookfor an unusual number of connections to or from an entity instance.

The risk trigger checks the number of connections from the time window it is analyzingagainst a baseline of activity from the previous 120 days.

For example, you could look for an account that has attempted to log into an unusually highnumber of machines. Or you could look for an IP address that has connected to an unusuallyhigh number of other IP addresses.

The configuration for an entity connections - degree spike risk trigger includes:

• The entity that receives the risk

• The specific relationship and direction for the analysis

• The time interval to use for the analysis.

For example, while the original time series may be configured with a 15-minute timeinterval, the risk trigger might analyze the data over hourly or daily time intervals.

• An optional WHERE clause to limit the relationship instances being analyzed

New built-in relationships track when useragents are used to request IP addresses andDNS domains

The CounterOps model now includes two new relationships that track when user agents areused to request IP addresses and DNS domains:

• DNSDomain requestedWith> UserAgent

• IPAddress requestedWith> UserAgent

The relationships are mapped from the PROXY_EVENT event type.

• DNSDomain requestedWith> UserAgent is mapped from the DST_DOMAIN and USER_AGENTglobal fields.



• IPAddress requestedWith> UserAgent is mapped from the DST_IP and USER_AGENT globalfields.

Enhancements to the Sqrrl query language andthe Sqrrl shell

New option to change the time interval when queryingtime series

When querying time series data, it may be more convenient to see the results using a largertime interval than the one the time series is configured with.

For example, most of the time series in the CounterOps model are configured with a 15-minute time interval. You may instead want to see the data aggregated over hourly, daily, ormonthly time intervals.

To enable alternate time intervals for time series, we:

• Added a new ts_downsample() query function.

The function returns a specified time series using the provided time interval and,optionally, a different aggregation method.

• Added a time interval parameter to the flatten_relationships() query function

• Added aggregation method and time interval parameters to the ts_flatten() queryfunction



New source abstraction layer to more easilymap and query source data

What is the source abstraction layer?

Many raw log files and feeds contain similar types of records and pieces of information.These records and fields are used to populate the same entities and relationships.

Previously, there was no way to map from or query the same types of records and fieldsacross sources. Each source was always mapped separately.

For 2.8, we've introduced a source abstraction layer, which consists of:

• Global fields, to identify specific pieces of information available from sources

• Event types, to identify types of activity reflected in source records

To see the list of global fields and event types, see the Reference Guide to Sqrrl Enterprise.

About global fields

Sqrrl global fields identify specific pieces of information that may be available from multiplesources.

You can map fields from any source to global fields, to identify how to use data from thesource to populate the global fields. For details on mapping sources to global fields, seeConfiguring and Loading Sqrrl Enterprise Data.

Many mappings are straight field-to-field mappings. For example, the SRC_IP global fieldmay be populated directly from a SourceIP field in a source.



Other mappings may be more complex. For example, a global field value may be a subset ofa source field value, or may combine values from multiple source fields.

About event types

Sqrrl event types identify specific types of activity, such as proxy requests or DNS serverresponses, that may be reflected in source records.

Each event type is associated with a list of global fields. Global fields may be used by morethan one event type, and are associated with at least one event type.

You can map any source to one or more event types, to indicate when a source recordcontains data for an event type. For details on mapping sources to event types, seeConfiguring and Loading Sqrrl Enterprise Data.

For some sources, every record contains data for the same event type. For example, allrecords from a proxy source apply to the proxy request event type.

Other sources may contain different types of records. In that case, you provide an expressionto identify when a record contains data for each applicable event type. For example, in amixed source, records are WINDOWS authentication records if the value of eventid is either4624 or 4625, but are DNS request records if there is a value in the questiontype field.



Using global fields and event types in SELECT queries

You can use event types and global fields in SELECT queries. Sqrrl then returns data from allsources that are mapped to those event types and global fields.

For example, the following query retrieves the source IP, destination IP, and destination portglobal field values from source records that are mapped to those fields, and that apply tothe FLOW_EVENT event type:

SELECT SRC_IP,DST_IP,DST_PORT FROM Global.FLOW_EVENT

For information on using event types and global fields in SELECT queries, see Exploring andQuerying Sqrrl Enterprise Data.

How the built-in Sqrrl source connectors use globalfields and event types

The built-in Sqrrl source connectors are all now mapped to event types and global fields. Forexample, for the Sqrrl_Netflow source connector:



For details on the source connector mappings, see Configuring and Loading Sqrrl EnterpriseData.

How the CounterOps model uses global fields and eventtypes

In the CounterOps model, we replaced most of the mappings from source connectors withmappings from Sqrrl event types and global fields.

The extraction process then automatically finds the sources that are mapped to those eventtypes and global fields. If you map your own sources to Sqrrl's event types and global fields,then Sqrrl automatically uses those sources to populate the CounterOps model.



How Sqrrl detectors use global fields and event types

We also mapped the source-based Sqrrl detectors to event types and global fields, instead ofto source connectors and source connector fields.

The detectors use any source mapped to the required event types and global fields. Ifyou map your own sources to those event types and global fields, then the detectorsautomatically use that data.

For each event type, you can also set up a list of excluded sources. The detector thenignores those excluded sources.

For example, by default, the Sqrrl_ProxySG source connector is excluded from theFLOW_EVENT event type for the Beacon detector, which is the required event type fordetecting IP-based beacons. So even though Sqrrl_ProxySG is mapped to the FLOW_EVENTevent type, it is not used to detect IP-based beacons.

For details on the event types and global fields used by the detectors, see Configuring andLoading Sqrrl Enterprise Data.

New query-based risk triggers allow for morecustomized detection of interesting activity

About risk triggers

Previously, all risk on entity instances such as IP addresses and URLs was based purely ontheir associated detection results.



For Sqrrl 2.8, we are introducing the concept of risk triggers, which you create and configurefrom the exploration list.

The query bar includes a new option to create risk triggers, which is only enabled forthe CounterOps model. The risk triggers are in the same list as the saved queries andinvestigations. For risk triggers, the list includes when the risk trigger last ran, and thenumber of matching entity instances from that run. From the list, you can enable, disable,edit, or delete a risk trigger.

You can create risk triggers for almost any entity or relationship in the CounterOps model,including entities that are not affected by detection results. You cannot create risk triggersfor detection results, relationships with detection results, or the Alert entity.

Risk triggers automatically watch the model to find entities that match user-specified testsfor suspicious activity. They can provide additional context for Sqrrl detection results, or canbe used to search for other activity not covered by the Sqrrl detectors.

You can use risk triggers to look for a wide range of behavior or activity, such as:

• Did an IP address connect to a suspicious web site from a threat intelligence feed?

• Did a user account log in over 100 times to the same server within a 3-minute window?

For details on creating and managing risk triggers, see Exploring and Querying SqrrlEnterprise Data.

Types of risk triggers

A risk trigger can:

• Find entity instances that match a filter, based either on entity features or relationshipvalues

• Check for rate spikes in a feature or relationship time series

How risk triggers are processed

Risk triggers run daily on one day's worth (by event time) of data at a time.



When the risk trigger query finds a match, Sqrrl contributes risk to the appropriate entityinstances.

How risk from risk triggers is stored

The risk from risk triggers is stored in a new risk framework, which consists of:

• Evidence sets, which store matches from risk triggers, including the event datetime

• Risk links, which link the matching entity instances to the corresponding evidence in theevidence set

The risk score for an entity instance is a normalized value based on both its associateddetection results and matches from risk triggers.

Redesigned risk dashboard provides high-level overview of both detector and risk triggerresults

The risk dashboard, previously called the detections dashboard, provides a high-level visualoverview of risk-related activity occurring during a selected time window.

We updated this page to reflect activity from both the built-in Sqrrl detectors and thenew user-controlled risk triggers. We also made changes to significantly improve the pageperformance.

The changes include:



• Limiting the available time windows. You can now view a summary of risk from either thelast day, last 7 days, last 30 days, or last 90 days.

• Changing the summary risk scores for each detection result type to be the highest scorefor a detection result with activity during the time window, instead of aggregating scoresacross results

• Adding a summary chart showing the top risk triggers for the time window, based on thenumber of matches

• Updating the Entities list to include entity instances affected by either detection resultsor risk triggers

• Adding a new Entities Identified by Triggers list to focus on entity instances affected byrisk triggers

• Streamlining the lists of detection results and entity instances

New risk timeline on entity instance profiletraces the history of an entity instance's risk

About the risk timeline

The risk for an entity instance changes over time based on multiple factors, including when:

• The entity instance is involved in new detection results

• An analyst classifies a detection result the entity instance is involved in

• The entity instance matches a risk trigger

• Specific amounts of time pass since a match from a detection result or risk trigger

On an entity instance profile, the new risk timeline provides a complete overview of therisk contributions from risk triggers and detection results for the currently selected timewindow.



For entity instances in the cybersecurity analytics model, the risk timeline replaces theHistory list from the right panel. The History list still displays on the exploration graphdetails panel, on profiles for detection results, and on profiles for entity instances in othermodels.

Viewing the changes in risk intensity over time, alongwith matches from risk triggers

For all entities, the risk timeline includes a risk heat map, which shows the relative intensityof risk over time, based on risk contributions from both detection results and risk triggers.

For each time interval, the risk is calculated based on activity from the previous 24 hours.Detection results and risk trigger matches before that have no effect.

The heat map also marks time intervals when the entity instance matched one or more risktriggers.

Hovering over a marked time interval displays the time interval, plus an indicator of thenumber of risk trigger matches. You can then display additional details about the risk triggermatches.



Viewing contributions from detection results to theentity instance

For instances of entities that can receive risk from detection results, the chart at the bottomof the timeline shows when detection results affected the risk score.

Each row in the chart is dedicated to a specific type of detection result. The chart does notinclude false positive or dismissed results.

Hovering over a bar in the detection chart displays the list of detection results reflected bythat bar.

You can then display the profile for a detection result, or jump to an exploration graphcontaining the listed results.

New Sqrrl Flume receivers and Flume flowsto load data directly from Netflow and IPFIXfeeds, support Sqrrl integrations

Sqrrl now comes with a built-in version of Flume. You can then designate one or more ofyour cluster nodes as Flume receivers.

Sqrrl maintains a list of the Flume receiver nodes. This list is similar to the lists for rolessuch as Sqrrl master, Sqrrl servers, monitors, and tracers.

Sqrrl also comes with built-in flows that you can assign to the Flume receivers. The built-inflows include flows for:

• Processing incoming netflow data that uses the following standard formats:

• Netflow 5

• Netflow 7

• Netflow 9

• IPFIX



These feeds all contribute data to a new built-in load job for the Sqrrl_Netflow sourceconnector. Once the flow is established, you simply point your export tool to the Flumereceiver and port.

• For the Sqrrl ArcSight integration, processing data exported from ArcSight to Sqrrl.

This replaces the Flume instance that was previously installed with the integration.

• For the Sqrrl QRadar integration, processing data exported from QRadar to Sqrrl.

This replaces the Flume instance that was previously installed with the integration.

New support for data from threat intelligencefeeds

New source connector to store threat intelligenceinformation

Involvement in a threat intelligence alert provides another source of evidence whenhunting for malicious activity. To help you use this data in your Sqrrl threat hunts, we areintroducing a new Sqrrl_ThreatIntel source connector, designed to contain data from threatintelligence feeds.

To accommodate different threat intelligence sources, the source connector contains a verybasic set of fields:

• The start and end datetimes when the indicator was seen

• The source of the indicator. This identifies the feed that provided the information.

• The type of indicator (IP address, URI, domain)

• The value of the indicator

• A string field containing any additional details

It also does not include a built-in load job or automatically created load directory. You canpopulate the Sqrrl_ThreatIntel source connector using any type of load job.

New feature to track the most recent threat intelligencehits on IP addresses, URIs, and DNS domains

We also added a new feature to track threat intelligence references for IP addresses, URIs,and DNS domains.



The lastIntelHit feature of the IPAddress, URI, and DNSDomain entities shows, for eachtype of threat intelligence feed, the date and time of the most recent alert for the entityinstance.

The Sqrrl_ThreatIntel source connector is mapped to the lastIntelHit feature. You canalso map your own sources to it.

New option to automatically increase a sourceshard count

As your Sqrrl cluster grows and more data is loaded, you may need to increase the shardcount of sources in order to accommodate it.

From the Sqrrl web application, you can now configure each source to have Sqrrlautomatically increase the shard count based on the history of loaded data for the source.Having the shard count increase automatically keeps your system optimized, and saves youfrom having to remember to make manual adjustments.

The new option is on the Change Shards dialog, displayed using the Change Shards buttonon the source summary page.



Other enhancements to the Sqrrl webapplication

New exploration graph actions menu provides easieraccess to graph options

In previous versions of Sqrrl, the exploration graph context menu contained many of theoptions for navigating and changing the scope of the exploration graph, including:

• Selecting and hiding entity instances

• Arranging selected entity instances

• Expanding the graph to add connected entity instances

• Drilling down to contributing source records

• Displaying the investigation panel

To make it easier to get to these options, we added a new graph actions menu, displayedusing a new graph actions icon on the graph toolbar.

The new graph actions menu contains all of the options that were originally in the contextmenu. It also contains the custom expansion option, which was previously an icon on thetoolbar.

The graph context menu now contains a reduced, single-level set of options to:

• Select and hide entity instances

• Expand the graph



Streamlined graph expansion options for theCounterOps model

The CounterOps model includes several relationships, many of which are closely connected.Instead of having to expand by one relationship at a time, the CounterOps model expansionoptions now allow you to expand the exploration graph based on relationship categories.

Each built-in relationship is assigned to a relationship category - a set of relatedrelationships. For example, the Detections and alerts category includes all of the involvedrelationships, and the Connections category contains the connectedTo and connectionFailedrelationships.

When you select a relationship category, Sqrrl expands the graph using all of therelationships in the category. For example, expanding an entity instance using theDetections and alerts category immediately shows you all of the connected detectionresults.

For models other than CounterOps, you still expand using individual relationships.



To expand using a single CounterOps relationship, you use the custom expansion option(Expand > Expand...).

For details on the relationships contained in each relationship category, see the informationon expanding an exploration graph in Exploring and Querying Sqrrl Enterprise Data.

Moved tags to a more prominent location on detectionresult and entity instance profiles

On profiles, the tags list has been moved from the details panel on the right to immediatelybelow the instance identifier at the top.

This allows you to quickly see what tags have been assigned to an entity instance ordetection result.

Enhancements to the Sqrrl shell

Relationship list now includes the origin and destinationentities

The results of the listrelationshipclass command now include the origin and destinationentities for the relationship, in addition to the UUID and name.

This helps to distinguish among relationships that have the same name, but differentendpoints. For example:

sqrrl:test@sqrrl> listrelationshipclass 2-2c: addedOrChanged : Trader > Security14-g: authenticationFailure : Account > Hostname14-h: authenticationFailure : Account > IPAddress14-i: authenticationSuccess : Account > Hostname14-j: authenticationSuccess : Account > IPAddress14-k: connectedTo : IPAddress > IPAddress14-l: connectionFailed : IPAddress > IPAddress



Including the job type in lists of job configurations andjob instances

Job configuration list now includes and is sorted by the job type

In the listjobconfigs command results, each job configuration starts with the job type.The results are sorted first by job type, and then alphabetically by job name:

sqrrl:test@sqrrl> listjobconfigs3-1z: [Detector] Data staging detection - prediction3-20: [Detector] Data staging detection - training3-2C: [Detector] DGA - prediction for Microsoft DNS Debug Logs3-2B: [Detector] DGA - training for Microsoft DNS Debug Logs3-28: [Detector] DNS Tunnel - prediction for Microsoft DNS Debug Logs3-27: [Detector] DNS Tunnel - training for Microsoft DNS Debug Logs3-23: [Detector] Exfiltration detection - prediction3-24: [Detector] Exfiltration detection - training3-1w: [Detector] Lateral Movement - prediction, multi-hop3-1v: [Detector] Lateral Movement - prediction, single login3-1u: [Detector] Lateral Movement - training3-1r: [Detector] Malicious beacon detection - netflow3-1q: [Detector] Malicious beacon detection - ProxySG URLs6-2: [Extension] ArcSight Beacon Export Job6-3: [Extension] ArcSight Data Staging Export Job6-4: [Extension] ArcSight Exfiltration Export Job6-1: [Extension] ArcSight Lateral Movement Export Job6-5: [Extension] ArcSight Risky Entity Export Job3-1s: [Extension] QRadar Beacon Export Job3-21: [Extension] QRadar Data Staging Export Job3-2D: [Extension] QRadar DGA Export Job3-29: [Extension] QRadar DNS Tunnel Export Job3-25: [Extension] QRadar Exfiltration Export Job3-1x: [Extension] QRadar Lateral Movement Export Job3-R: [Extension] QRadar Offense Pre-process Job3-j: [Extension] QRadar risky entity export job6-1p: [Load] Alerts Load Job3-O: [Load] Alerts load job for ArcSight3-2G: [Load] Alerts load job for Preprocess Datasource3-S: [Load] Alerts load job for QRadar...

Job instance list now includes the job type

In the listjobs command results, each job instance now includes the job type:

sqrrl:test@sqrrl> listjobs[Load 3y SUCCEEDED] ProxySG load job: "No new records to process. Job successfully completed."[Load 3v SUCCEEDED] Windows Event Log load job: "No new records to process. Job successfully completed."[Load 3u SUCCEEDED] Netflow load job: "No new records to process. Job successfully completed."[Load 3r SUCCEEDED] Microsoft DNS Debug log load job: "No new records to process. Job successfully completed."[Detector 1P SUCCEEDED] Data staging detection - prediction: "Finished spark job"[Detector 1M SUCCEEDED] Exfiltration detection - training: "Finished spark job"[Detector 1L SUCCEEDED] Exfiltration detection - prediction: "Finished spark job"[Detector 13 SUCCEEDED] Exfiltration detection - prediction: "Finished spark job"[Detector y SUCCEEDED] Data staging detection - training: "Finished spark job"[RiskTrigger u SUCCEEDED] Account with many failed logons: "Finished spark job"[RiskTrigger t SUCCEEDED] Requested suspicious URI: "Finished spark job"[Detector s SUCCEEDED] Lateral Movement - prediction, multi-hop: "Finished spark job"[Detector r SUCCEEDED] Lateral Movement - prediction, single login: "Finished spark job"[Detector q SUCCEEDED] Lateral Movement - training: "Finished spark job"[Detector n FAILED] DGA - training for Microsoft DNS Debug Logs: "Job failed with code 1"[Detector g SUCCEEDED] DNS Tunnel - prediction for Microsoft DNS Debug Logs: "Finished spark job"



[Detector M SUCCEEDED] Malicious beacon detection - netflow: "Finished spark job"[Detector L SUCCEEDED] Malicious beacon detection - ProxySG URLs: "Finished spark job"

New option to view query function help from the Sqrrlshell

When issuing queries from the Sqrrl shell, you often need to use query functions, some ofwhich have complex syntax or rules for use.

Previously, the help for query functions was only available from the Reference Guide forSqrrl Enterprise.

In the Sqrrl shell, the help command by default displays the help for Sqrrl shell commands.

help commandName

For example:

sqrrl:test@sqrrl> help testvisrulesUsage: testvisrules [-?] -j <jsonFile> [-p] -rf <rulesFile>Description: Used to test how a set of visibility rules would be applied to actual data.

When issuing the command, you provide both the JSON file containing the data, and the text file containing the visibility rules.

The command returns the same data with visibility labels added based on the provided visibility rules.

-?,--help Display this help. -j,--json-file <jsonFile> The file containing the data to use to test the visibility rules. The file contains a single JSON object.-p,--plain If included, then the results are displayed in a single line, without additional formatting such as line breaks and indents.-rf,--rules-file <rulesFile> The file containing the visibility rules to test.

We've enhanced the help command to allow you to also display help for query functions. Todisplay the help for a query function, use the -q switch.

help -q queryFunctionName

For example:

sqrrl:test@sqrrl> help -q dayofweekUsage: dayofweek(datetime)

Returns the day of the week represented by the datetime.

The value is in the range 1-7, with 1 for Monday through 7 for Sunday.

For example, the following SELECT query returns log records that were created on a Tuesday:

SELECT SourceAddress,TimeCreated FROM FlowLogs WHERE dayofweek(TimeCreated) = 2

Pagination control extended to all Sqrrl shell andadministrative shell commands

By default, Sqrrl shell command results are paginated, meaning that initially, the commandonly returns as much as will fit on the current screen. You can then press any key to displaythe next page of results, or press q to exit back to the shell prompt.



---------- To continue, press any key. To quit, press 'q'. ------------

In some cases, however, you may not want the results to be paginated. For example, whenretrieving a configuration object, you don't want the results interrupted with a pagingprompt.

The -np option indicates to not paginate the command results.

Previously, the -np option was only available on subset of commands.

The -np option is now available for all shell and administrative shell commands.

You can also provide the -np option when you start the Sqrrl shell or administrative shell, todisable pagination for all commands issued during that shell session.

New option to edit configuration objects directly fromthe Sqrrl shell command line

Previously, the only way to edit configuration objects from the Sqrrl shell was to:

1. Use the getitemType command to save the configuration object JSON to a file.

2. Edit the JSON file.

3. Use the updateitemType command to upload the updated configuration object JSON toSqrrl.

The new -i option on the updateitemType command indicates to open the configurationobject JSON in a command line-based text editor.

updatedatasource A-4 -i

If possible, the configuration object JSON is opened using the editor set as the value of theEDITOR environment variable. If that variable isn't set, or isn't set to a command line-basededitor, then the configuration object JSON is opened in Vim.

When you finish editing and save the changes, the updated JSON is automatically uploaded.

New automatic detection of the UUID to update for theupdate command

When using the updateconfigobject or updateitemType command to update a configurationobject, if the JSON file includes the UUID of that object, then you do not need to specify itin the command.



New name/template ID macros for references to otherconfiguration items

A configuration item may need to refer to another configuration item. For example, sourceload jobs need to identify the source they are loading data into, and entities need toidentify the model they belong to.

In the configuration JSON, the property for identifying the item is usually itemTypeUuid. Forexample, datasourceUuid or modelUuid.

Previously, you always had to specify the actual UUID. Now, to make it easier to manage thereferences, and to port items between clusters, you can instead use a name or template IDmacro. Sqrrl then uses the macro to determine the correct UUID.

The syntax for a name macro is ${itemName}. For example, ${HostLogs}.

For built-in items, the syntax for a template ID macro is ${templateId:templateIDValue}.For example, ${templateId:ProxySgDatasource}.

When you use the getconfigobject or getitemType command to retrieve a configurationitem, references to other configuration items by default use the object UUID.

sqrrl:test@sqrrl> getentityclass 20-S -j{ "uuid" : "20-S", "name" : "File", "description" : "A file, either on disk or in transit.", "createdAt" : 1488487975866, "updatedAt" : 1488487975866, "lastUpdatedBy" : "sqrrl", "version" : 0, "formatVersion" : 1, "templateId" : "File", "visualizationOptions" : { "color" : "#808080", "image" : "mono--file" }, "modelUuid" : "20-2", ...

To display the name or template ID macro instead, use the -ref parameter.

sqrrl:test@sqrrl> getentityclass 20-S -j -ref{ "uuid" : "20-S", "name" : "File", "description" : "A file, either on disk or in transit.", "createdAt" : 1488487975866, "updatedAt" : 1488487975866, "lastUpdatedBy" : "sqrrl", "version" : 0, "formatVersion" : 1, "templateId" : "File", "visualizationOptions" : { "color" : "#808080", "image" : "mono--file" }, "modelUuid" : "${templateId:CounterOpsModel}", "features" : [ {...



If you include the --config-by-ref option when you start the Sqrrl shell, then allreferences automatically display using the name or template ID macro.

New compact display format for time series values

For MATCH query results in the Sqrrl shell, we have introduced a new compact table formatfor time series values. The new format allows you to see more of the actual values.

Here is a MATCH query that includes a time series feature in the results:

In this compact format, the time interval length displays at the top right of the table.

Each row begins with the timestamp for the first value in the row. For gaps - time intervalsthat do not have a value:

• If the gap is short enough, the row displays a dot for each empty time interval, followedby the value for the next populated time interval.

The row does not include timestamps for these later populated time intervals.

• If the gap is too long, then a new row is started.

The new row begins with an ellipsis (...), followed by the timestamp for the nextpopulated time interval.

The compact format is the default for time series values. To instead use the previous treestructure for time series, add the --ts-format=full option.

MATCH User{kjackson}[**] FROM SecurityAnalysis --ts-format=full



DNS-related source connector and detectorsadded to the Sqrrl hunting tool set

New source connector to store Microsoft Server DNSDebug entries

The new Sqrrl_MSDNSDebug source connector is used to load data from MS Server 2008 and2012 DNS Debug logs.

The new source connector comes with a built-in load job and automatically created loaddirectory.

The Sqrrl_MSDNSDebug source connector is mapped to the DNSDomain and IPAddress entitiesin the cybersecurity analytics model, and provides the source data for the new DNS-relateddetectors.

For details on the Sqrrl_MSDNSDebug source connector and load job configuration, seeConfiguring and Loading Sqrrl Enterprise Data.

New detector to find instances of DNS tunneling

Sqrrl's new DNS tunnel detector analyzes data from the new Sqrrl_MSDNSDebug sourceconnector for evidence of DNS tunneling, where a series of DNS requests to the sameregistered domain have data embedded in the subdomain.

A DNS tunnel detection result is linked to the origin IP address and the registered domain forthe suspicious requests.



For details on configuring the DNS tunnel whitelist, and the DNS tunnel detector processing,see Configuring and Loading Sqrrl Enterprise Data.

For details on analyzing profile data for DNS tunnel detection results, see Exploring andQuerying Sqrrl Enterprise Data.

New detector to find instances of domain generationalgorithm activity

The new DGA (domain generation algorithm) detector analyzes data from the newSqrrl_MSDNSDebug source connector for evidence of domain generation algorithm activity.Attackers use the algorithm to generate domain names, then use them to make DNS requestsfrom the infected workstation.

Most of these requests fail with a "domain does not exist" error. However, when attackersare ready to send additional instructions to the infected machine, they assign a generateddomain name to their command and control server.

A DGA detection result is linked to the origin IP address for the domain resolution requests,and the domains it attempted to connect to.



For details on configuring the DGA whitelist, and the DGA detector processing, seeConfiguring and Loading Sqrrl Enterprise Data.

For details on analyzing profile data for DGA detection results, see Exploring and QueryingSqrrl Enterprise Data.

Built-in whitelist of common registered domains reducesfalse positives for DNS-related detectors

To reduce the number of false positives for DNS-related detectors, and also reduce theburden of whitelisting known domains, Sqrrl comes with a large built-in whitelist of commondomains. The list is from the Alexa top 1000 domain list.

On the detector details panel for the DNS tunnel and DGA detectors, the Ignore well-knownregistered domains checkbox determines whether the detector uses the built-in whitelist.

By default, the built-in domain whitelist is enabled.

For details on controlling whether to use the built-in whitelist for the DNS tunnel and DGAdetectors, see Configuring and Loading Sqrrl Enterprise Data.



New built-in report elements for DNS data

To provide additional insight into your DNS request data, we've added new report elementsto the built-in Hunt Reports.

These new report elements highlight information about the domains found in the DNS data.

Search and query enhancements

New combined search field speeds up searches for tagsand instance IDs

The search field at the top of the Sqrrl web application is now dedicated to searches foreither:

• A specific entity instance ID

• A tag that is assigned to entity instances

As you type, a type-ahead list shows both matching instance IDs and tag names in thecurrently selected model. The tag search is case insensitive, while the instance ID search iscase sensitive.



To start the search, and display the search results page, either press Enter or select amatching value.

The search results page displays the list of instances with an exact match for a tag name orinstance ID.

For details on completing tag and instance ID searches, see Exploring and Querying SqrrlEnterprise Data.

New expanded query panel for the Explore context

In the Sqrrl web application, the Explore context includes:

• SELECT query results

• The Explore graph

• The list of saved queries and investigations

When in the Explore context, a new query panel now displays at the top of the page. Theexpanded query text area displays the most recent query that was issued for the currentlyselected model.

From the query panel, you can:

• Run the currently displayed query

• Enter and run a new query



• Save the currently displayed query

• For SELECT results:

• Explore results on an Explore graph

• Export results to a CSV file

• For an Explore graph, the toolbar save menu has been removed, and the options havebeen moved to the query panel Save menu.

So from the query panel, you can:

• Save the current investigation

• Export the graph to a PNG file

• Export the list of displayed nodes to a CSV file

• Save the current outlier filter to a report item

For details on using the new query panel to issue queries and work with graphs and queryresults, see Exploring and Querying Sqrrl Enterprise Data.

Source connector, load jobs for SIEMintegration now built into Sqrrl

The Sqrrl ArcSight integration uses:

• The Sqrrl_Alerts source connector, designed to store alerts from SIEMs such as ArcSight.

The Sqrrl_Alerts source connector is mapped to the Alert entity.

• Load jobs specifically configured to load data via ArcSight SmartConnectors into Sqrrlsource connectors

Previously, these were installed with the integration itself. As of Sqrrl 2.7, they are builtinto Sqrrl. Installing the integration then installs the required plugin .jar file, creates therequisite load job directories, and automates the job schedules.



For the Sqrrl_Alerts source connector, even if you don't use a Sqrrl integration, you can addnew load jobs of any type in order to use the Sqrrl_Alerts source connector for your specificdata.

For details on the Sqrrl_Alerts source connector configuration, see Configuring and LoadingSqrrl Enterprise Data.

New ARRAY data type for source fields

Configuring an array source field

Sqrrl sources now support an ARRAY data type for source fields. A source array field mustcontain a list of values all of the same type, defined using the subtype setting.

{ "path": "tags", "type": "ARRAY", "subtype": "STRING", "description": "A list of tags assigned to the message."}

New fors element to map array values to entities andrelationships

In the entity and relationship JSON configuration, a new fors element allows you to loopthrough and map the individual array values.

For example, this fors element loops through arrays of IP addresses and hostnames in thesource records, then maps the individual values to the entity identifier and the hostnamefeature.

"fors": [ { "datasourceUuid": "3-K", "each": { "ip": "IPAddresses", "hostname": "Hostnames" }, "indexRef": "idx", "mappings": [ { "entityIdentity": "${ip}", "mappingEntries": { "Hostname": { "value": "${hostname}" } } } ] }]

For details on the JSON configuration for configuring and mapping source array fields, seethe Reference Guide for Sqrrl Enterprise.



New array_ref() query function to extract values fromarrays

A new array_ref() query function retrieves the value at the specified index of an array.The arrays are zero-based.

For example, array_ref('SitesVisited', 1), returns the second value in the SitesVisitedarray.

new and noteworthy in sqrrl enterprise - version 2.9 · new and noteworthy in sqrrl enterprise 5...

Documents